CN112732738B - Adaptive network data acquisition method based on multi-objective optimization and related equipment - Google Patents
Adaptive network data acquisition method based on multi-objective optimization and related equipment Download PDFInfo
- Publication number
- CN112732738B CN112732738B CN202110210085.6A CN202110210085A CN112732738B CN 112732738 B CN112732738 B CN 112732738B CN 202110210085 A CN202110210085 A CN 202110210085A CN 112732738 B CN112732738 B CN 112732738B
- Authority
- CN
- China
- Prior art keywords
- acquisition
- data
- equipment
- acquired
- frequency
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/12—Computing arrangements based on biological models using genetic models
- G06N3/126—Evolutionary algorithms, e.g. genetic algorithms or genetic programming
Abstract
The embodiment of the specification provides a self-adaptive network data acquisition method based on multi-objective optimization and related equipment; the method comprises the following steps: predicting data to be collected by utilizing a Holt-Winters method; quantizing the frequency and the acquisition distortion of the acquired data, and constructing a multi-objective optimization problem; converting the multi-objective optimization problem into a new objective function through a target weighting method, and solving the new objective function by using a genetic algorithm based on the prediction data to obtain an optimal acquisition time sequence; and dynamically adjusting the weight parameters in the new objective function according to the CPU utilization rate of the adopted equipment. Compared with the traditional periodic acquisition method, the method provided by the specification considers the change of the acquired data, reduces the frequency of the acquired data by reasonably distributing acquisition time points, reduces the distortion caused by the acquisition process as much as possible, takes the CPU utilization rate of the acquired equipment into consideration range, and avoids causing overlarge acquisition burden on the equipment.
Description
Technical Field
One or more embodiments of the present disclosure relate to the field of digital information transmission and data acquisition technologies, and in particular, to an adaptive network data acquisition method and related device based on multi-objective optimization.
Background
In the current internet application scenario, data becomes more and more important, and the data is a basis for supporting implementation of many services, for example, network data is collected, and then characteristics of the data are analyzed to detect attacks and intrusions, so as to implement protection of a network system, but at present, data collection is a performance bottleneck of most of service systems related to data.
The data acquisition algorithm with equal time intervals adopted in a large amount of existing network management software periodically acquires data of managed equipment according to acquisition frequency set by a system or a user, and changes of the acquired data are ignored. For example, when the acquired data changes violently, the change trend of the acquired object cannot be accurately reflected due to too low acquisition frequency, and important data is easily lost; when the acquired data is basically unchanged, if the acquisition frequency is too high, data redundancy and network resource waste are caused, and meanwhile, because the requested data volume is large, a Simple Network Management Protocol (SNMP) agent of the equipment can not respond to the request, even occupies too high a Central Processing Unit (CPU), and other normal services of the network equipment are influenced.
Based on this, a method capable of adaptively acquiring network data is needed.
Disclosure of Invention
In view of this, an object of one or more embodiments of the present disclosure is to provide an adaptive network data acquisition method and related device based on multi-objective optimization.
In view of the above, one or more embodiments of the present specification provide an adaptive network data acquisition method based on multi-objective optimization, including:
data acquisition is carried out on the equipment according to an acquisition time schedule, and the acquired data are stored in a database;
based on historical collected data in the database, calculating a predicted value of the collected data in the next time period by using a Holt-Winters method;
based on the predicted value, establishing a multi-objective optimization problem by taking the frequency and the collection distortion of collected data as optimization targets, then obtaining a new objective function by a target weighting method, and solving the new objective function by using a genetic algorithm to obtain an optimal collection time sequence;
when the update threshold of the acquisition time schedule is reached, calling the optimal acquisition time sequence to update the acquisition time schedule;
and when the CPU utilization rate of the equipment is smaller than an idle threshold or larger than a busy threshold, adjusting the weight parameter in the new objective function to change the frequency of the acquired data.
Based on the same inventive concept, one or more embodiments of the present specification further provide an adaptive network data acquisition apparatus based on multi-objective optimization, including:
the acquisition module is configured to acquire data of the equipment according to an acquisition time schedule and store the acquired data into a database;
the prediction module is configured to calculate a predicted value of the acquired data in the next time period by utilizing a Holt-Winters method based on historical acquired data in the database;
the calculation module is configured to construct a multi-objective optimization problem by taking the frequency of acquired data and the acquisition distortion degree as optimization targets based on the predicted values, then obtain a new objective function through a target weighting method, and solve the new objective function by using a genetic algorithm to obtain an optimal acquisition time sequence;
an update module configured to invoke the optimal acquisition time sequence to update the acquisition schedule when an update threshold of the acquisition schedule is reached;
an adjusting module configured to adjust a weight parameter in the new objective function to change a frequency of acquiring data when a CPU utilization of the device is less than an idle threshold or greater than a busy threshold.
Based on the same inventive concept, one or more embodiments of the present specification further provide an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the method as described in any one of the above items when executing the program.
Based on the same inventive concept, one or more embodiments of the present specification also provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in any one of the above.
As can be seen from the foregoing, in the adaptive network data acquisition method and the related device based on multi-objective optimization provided in one or more embodiments of the present specification, changes of acquired data are considered, and a genetic algorithm based on target weighting is used to minimize the frequency of acquired data and the acquisition distortion, so that the frequency of acquired data is reduced, distortion caused by an acquisition process is reduced as much as possible, the accuracy of acquired data is improved, and meanwhile, the CPU utilization of the acquired device is taken into consideration, thereby avoiding an extra performance burden of the device due to acquisition.
Drawings
In order to more clearly illustrate one or more embodiments or prior art solutions of the present specification, the drawings that are needed in the description of the embodiments or prior art will be briefly described below, and it is obvious that the drawings in the following description are only one or more embodiments of the present specification, and that other drawings may be obtained by those skilled in the art without inventive effort from these drawings.
FIG. 1 is a flow diagram of a method for adaptive network data acquisition based on multi-objective optimization according to one or more embodiments of the present disclosure;
FIG. 2 is a schematic structural diagram of an adaptive network data acquisition device based on multi-objective optimization according to one or more embodiments of the present disclosure;
fig. 3 is a schematic structural diagram of an electronic device according to one or more embodiments of the present disclosure.
Detailed Description
For the purpose of promoting a better understanding of the objects, aspects and advantages of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.
It is to be noted that unless otherwise defined, technical or scientific terms used in one or more embodiments of the present specification should have the ordinary meaning as understood by those of ordinary skill in the art to which this disclosure belongs. The use of the terms "comprising" or "including" and the like in one or more embodiments of the present specification is intended to indicate that an element or item preceding the term includes the element or item listed after the term and its equivalent, but not to the exclusion of other elements or items.
As described in the background section, the existing network data acquisition method ignores the change of the acquired data, thereby causing the problems of large acquisition distortion, inaccurate acquired data, increased equipment acquisition burden and the like. At present, some techniques propose adaptive acquisition to dynamically adjust the frequency of acquired data, which dynamically adjusts the frequency of acquired data according to the changing situation of the acquired object.
The applicant finds that the existing adaptive network data acquisition method has some defects in the process of implementing the disclosure: the acquisition frequency is adjusted according to the recent data fluctuation degree, the change of an acquisition object is calculated in real time, and the calculation consumption is large and the accuracy is insufficient; adjustment of the acquisition frequency is usually performed by setting an adjustment rule according to the data fluctuation degree by subjective experience, so that the adjustment scale is difficult to accurately grasp, the acquisition efficiency or the acquisition precision is easy to reduce, and the reliability is poor; the influence of the collection behavior on the collected equipment is not considered, so that the collected equipment is frequently collected when the pressure of the collected equipment is high, and serious consequences such as equipment downtime and the like are caused.
In view of this, one or more embodiments of the present disclosure provide an adaptive network data acquisition method based on multi-objective optimization, and specifically, data acquisition is performed on devices according to an acquisition schedule, and the acquired data is stored in a database, where the acquisition schedule stores the acquisition data time of each acquired device. And then, based on the historical collected data in the database, calculating by using a Holt-Winters method to obtain a predicted value of the collected data in the next time period. In addition, based on the predicted value, the acquisition strategy target is quantized into two optimization targets, namely, the frequency and the acquisition distortion of the acquired data are used as the optimization targets to construct a multi-target optimization problem, then a new target function is obtained through a target weighting method, and the new target function is solved through a genetic algorithm to obtain an optimal acquisition time sequence. Further, when the update threshold of the acquisition schedule is reached, the optimal acquisition time sequence is called to update the acquisition schedule. And finally, using the CPU utilization rate of the collected equipment as a measurement standard, and dynamically adjusting the weight parameter in the new objective function to change the frequency of the collected data when the CPU utilization rate of the equipment is smaller than an idle threshold or larger than a busy threshold.
Therefore, the adaptive network data acquisition method based on multi-objective optimization in one or more embodiments of the present specification considers the change of the acquired data, reduces the frequency of the acquired data, reduces the acquisition distortion as much as possible, improves the accuracy of the acquired data, and simultaneously takes the CPU utilization of the acquired device into consideration, thereby avoiding the extra and overlarge performance burden of the device due to acquisition.
The technical solutions of one or more embodiments of the present specification are described in detail below with specific examples.
Referring to fig. 1, an adaptive network data acquisition method based on multi-objective optimization in one embodiment of the present specification includes the following steps:
and S101, acquiring data of the equipment according to an acquisition time schedule, and storing the acquired data into a database.
In this step, an n-bit binary Y = { Y } is maintained for each device to be acquired in the acquisition schedule 1 ,y 2 ,…,y n-1 ,y n }，y i E is {0,1}, i is more than or equal to 1 and less than or equal to n, wherein y i (1. Ltoreq. I. Ltoreq.n) at t 0 Whether to collect at + i x Δ t moment, 1 represents collecting, 0 represents not collecting, and t represents collecting 0 As the current time, Δ t is the minimum acquisition time interval;
the step S101 specifically includes: traverse the acquisition schedule, y, every Δ t i If the number of the equipment is 1, adding the identification OID corresponding to the equipment into a list to be acquired, and if the number of the equipment is 0, not operating;
after traversing, constructing a Simple Network Management Protocol (SNMP) request according to the list to be acquired, and initiating a data acquisition request to each device;
and storing the acquired data into a database.
And S102, calculating a predicted value of the acquired data in the next time period by utilizing a Holt-Winters method based on the historical acquired data in the database.
The step S102 specifically includes:
an initial value part that selects, based on the historical collected data in the database, the historical data of the first two cycles using the first predicted point of the data to be collected next for calculation:
the initial value of the stable value is:
the initial value of the trend value is:
the initial seasonal value is:
wherein S i ' for each moment of time during the first cycle:
S i "is the seasonal value at each time in the second period:
wherein L is the cycle length, i.e. the number of acquisitions in each cycle, x i Historical data values of the first two periods of the first predicted point for the next data to be collected;
obtaining the stationarity value a of the data acquired by the equipment at the t time by recursion according to the calculation formula of the initial value part t Trend value b t And a seasonal value S t Wherein said a t 、b t 、S t The calculation formula of (2) is as follows:
b t ＝β(a t -a t-1 )+(1-β)b t-1
a above t 、b t 、S t In the calculation formula, the coefficients α, β and γ are independent variables which determine the predicted values of the performance index data corresponding to the respective stationarity, tendency and seasonality, and the value ranges thereof are all intervals (0, 1). In case of sufficient historical data, corresponding prediction can be made in advance through any combination of exhaustive 3 smoothing coefficients (from 0.01 to 0.99), according to which, respectively, corresponding prediction is made through calendarThe history record calculates to obtain the square sum of the relative error values, and selects the smoothing coefficient corresponding to the minimum error as the basic value of the optimal smoothing coefficient. During real-time prediction calculation, the collected data of the latest period is used as a predicted value, adjustment is carried out on a basic value, and alpha, beta and gamma with the minimum corresponding errors are selected;
predicting the performance index data predicted value x after h minimum acquisition interval durations by using Holt-Winters method t+h The prediction formula is:
x t+h ＝(a t +h*b t )S t+h-L
wherein S is t+h-L Can be composed of S t Calculated, h =1,2,3 \ 8230n, when h =1, x t+1 For the predicted value of the first time point to be acquired after 1 minimum acquisition interval duration, x is calculated when h =2 t+2 After 2 minimum acquisition interval durations, acquiring a predicted value of a second time point, and then calculating successively and repeatedly in a recursion way to obtain a predicted value sequence P of the next n time points, wherein P = { P = 1 ,p 2 ,…,p n-1 ,p n }，p i Is represented at t 0 The predicted value at the moment of + i Δ t, wherein i is more than or equal to 1 and less than or equal to n, and t 0 At the current time, Δ t is the minimum acquisition time interval.
And S103, based on the predicted value, constructing a multi-objective optimization problem by taking the frequency and the collection distortion of collected data as optimization targets, then obtaining a new objective function by a target weighting method, and solving the new objective function by using a genetic algorithm to obtain an optimal collection time sequence.
The step S103 specifically includes: setting the best acquisition time sequence of any equipment solved by a genetic algorithm based on a target weighting method as a binary number X with n bits, wherein X = { X = 1 ,x 2 ,…,x n-1 ,x n }，x i Is belonged to {0,1}, i is more than or equal to 1 and less than or equal to n, wherein x i Is shown at t 0 Whether collection is carried out at + i × Δ t moment, 1 represents collection, 0 represents non-collection, and t represents collection 0 At the current time, Δ t represents the minimum acquisition time interval;
will collect dataThe frequency of (d) is defined as:
the acquisition distortion factor is defined as:
wherein p is i Is represented at t 0 Predicted value at time + i Δ t, s i Representing the corresponding time point t on the fitting curve obtained by assuming that data are collected according to the X 0 And (3) taking the frequency and the collection distortion degree of the collected data as optimization targets according to the data values on + i × Δ t, thereby obtaining a multi-target optimization problem: minG (X) = (F (X), R (X));
then, solving the multi-objective optimization problem by using a genetic algorithm based on an objective weighting method, firstly weighting the multi-objective optimization problem by using the objective weighting method, namely, dividing each optimized objective vector into one weight to obtain a new objective function:
G(X)＝(1-w r )F(X)+w r R(X)
wherein, w r As weight parameter, 0 < w r ＜1；
And then, solving the new objective function by using a genetic algorithm to obtain an optimal acquisition time sequence X corresponding to each equipment minG (X).
The step of solving the new objective function by using the genetic algorithm comprises the following steps:
(a) And initializing to generate an initial population. Since the solved binary number is the binary number, the encoding operation is not needed, and the binary number with the length of n bits is the individual in the population. Firstly, initializing a population Q with a population scale of Q, namely randomly generating Q binary numbers with n bit lengths;
(b) Individual evaluation fitness evaluation was performed for each individual gene in the population Q using an adaptive function, i.e., the inverse number of G (X) described above, i.e., Q (X) = - (1-w) r )F(X)-w r R (X), and judging whether an optimization criterion condition K is met, namely Q (X) is larger than a preset target value Q max . If yes, the flow is interruptedGetting out the best individual and stopping; if the current evolution times reach the maximum times E max Outputting the individual with the highest adaptability in the population; otherwise, continuing the next step;
(c) According to the selection probability, a selection operator is executed, and partial individuals are selected from the current population to enter the next generation of population;
(d) Performing crossover operators according to crossover probabilities, wherein uniform crossover is used in the present embodiment, and genes at each locus of two paired individuals are exchanged with the same crossover probability;
(e) Executing mutation operators according to the mutation probability, wherein uniform mutation is used in the embodiment, and original gene values of all loci in the individual code strings are replaced by a certain smaller probability;
(f) And (c) generating a new generation of population by crossing and mutation, and returning to the step (b).
From this, the update value for each device, i.e. the optimal acquisition time series X, is found.
And step S104, calling the optimal acquisition time sequence to update the acquisition time table when the update threshold of the acquisition time table is reached.
The step S104 specifically includes: maintaining an n-bit binary number Y for each device to be acquired in the acquisition time table, reading a one-bit binary number at intervals of delta t, and when w (1-w-n) th bits are read, reaching an update threshold of the acquisition time table, wherein each device to be acquired obtains an optimal acquisition time sequence X;
replacing an n-bit binary number Y maintained for each device of the acquired data in the acquisition time table with the optimal acquisition time sequence, i.e., the n-bit binary number X;
and acquiring data of the equipment according to the updated acquisition time table.
And S105, when the CPU utilization rate of the equipment is smaller than an idle threshold or larger than a busy threshold, adjusting the weight parameter in the new objective function to change the frequency of the acquired data.
The step S105 specifically includes: setting an idle threshold value for each acquired data device as an idle time period CPU utilization rate U free Setting busy threshold as CPU utilization rate U in busy time period busy ；
Polling the device periodically when the CPU utilization U of the device cpu Is less than the U free Reducing the weight parameter w in the new objective function r Let w be r ＝w r (ii)/2, such that the frequency of said data acquisition is increased;
when U of the device cpu Greater than or equal to U free And is less than or equal to U busy The weight parameter w in the new target function number r The change is not changed;
when U of the device cpu Is greater than the U busy While increasing the weight parameter w in the new objective function r Let w be r ＝w r + Δ w, such that the frequency of the acquired data is reduced.
It can be seen that the adaptive network data acquisition method based on multi-objective optimization provided in the embodiments of the present specification quantifies the frequency and the acquisition distortion of the acquired data, constructs a multi-objective optimization problem with the frequency and the acquisition distortion of the acquired data as optimization targets, solves the multi-objective optimization problem by using a genetic algorithm based on target weighting, and calculates an optimal acquisition time sequence in a future time period. Compared with a method for calculating an acquisition frequency, the method provided by the specification avoids frequent calculation and is more flexible, and meanwhile, binary storage is adopted, so that excessive storage resources are avoided being occupied; compared with the traditional periodic acquisition method, the method provided by the specification considers the change of the acquired data, reduces the frequency of the acquired data and the acquisition distortion as much as possible, predicts the value of the acquired data in a future time period by using a Holt-Winters method, calculates the optimal acquisition time point based on the value, improves the accuracy of the acquired data, takes the CPU utilization rate of the acquired equipment into consideration range, and avoids extra and overlarge performance burden of the equipment caused by acquisition.
It should be noted that the method of one or more embodiments of the present disclosure may be executed by a single device, such as a computer or a server. The method of the embodiment can also be applied to a distributed scene and completed by the mutual cooperation of a plurality of devices. In such a distributed scenario, one of the multiple devices may perform only one or more steps of the method of one or more embodiments of the present description, and the multiple devices may interact with each other to complete the method.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Based on the same inventive concept, corresponding to any embodiment of the method, one or more embodiments of the present specification further provide an adaptive network data acquisition device based on multi-objective optimization. Referring to fig. 2, the adaptive network data acquisition device based on multi-objective optimization includes:
the acquisition module 201 is configured to acquire data of the equipment according to an acquisition schedule and store the acquired data in a database;
the prediction module 202 is configured to calculate a predicted value of the acquired data in the next time period by using a Holt-Winters method based on historical acquired data in the database;
the calculation module 203 is configured to construct a multi-objective optimization problem by taking the frequency of the acquired data and the acquisition distortion as optimization targets based on the predicted values, obtain a new objective function by a target weighting method, and solve the new objective function by using a genetic algorithm to obtain an optimal acquisition time sequence;
an update module 204 configured to invoke the optimal acquisition time sequence to update the acquisition schedule when an update threshold of the acquisition schedule is reached;
an adjusting module 205 configured to adjust weight parameters in the new objective function to change a frequency of acquiring data when a CPU utilization of the device is less than an idle threshold or greater than a busy threshold.
As an alternative embodiment, the collecting module 201 is specifically configured to maintain an n-bit binary number Y, Y = { Y } for each device to be collected in the collection schedule 1 ,y 2 ,…,y n-1 ,y n }，y i Belongs to {0,1}, i is more than or equal to 1 and less than or equal to n, wherein y i Is shown at t 0 Whether collection is carried out at + i × Δ t moment, 1 represents collection, 0 represents non-collection, and t represents collection 0 As the current time, Δ t is the minimum acquisition time interval; traverse the acquisition schedule, y, every Δ t i If the number of the equipment is 1, adding the identification OID corresponding to the equipment into a list to be acquired, and if the number of the equipment is 0, not operating; constructing a Simple Network Management Protocol (SNMP) request according to the list to be acquired, and initiating a data acquisition request to each device; and storing the acquired data into a database.
As an alternative embodiment, the prediction module 202 is specifically configured to calculate a stationarity value a of the data acquired by the device at the t-th time based on the historical acquired data in the database t Trend value b t And a seasonal value S t (ii) a Predicting the performance index data predicted value x after h minimum acquisition interval durations by using Holt-Winters method t+h The prediction formula is: x is the number of t+h ＝(a t +h*b t )S t+h-L Wherein S is t+h-L Can be composed of t Calculating to obtain; calculating a predicted value sequence P, P = { P) of the next n time points by the prediction formula 1 ,p 2 ,…,p n-1 ,p n }，p i Is represented at t 0 The predicted value at the moment of + i Δ t, wherein i is more than or equal to 1 and less than or equal to n, and t 0 Is as followsThe previous time, Δ t, is the minimum acquisition time interval.
As an alternative embodiment, the calculation module 203, is specifically configured to set the optimal acquisition time sequence of any equipment solved by a genetic algorithm based on a target weighting method to be n-bit binary number X, X = { X = 1 ,x 2 ,…,x n-1 ,x n }，x i Is belonged to {0,1}, i is more than or equal to 1 and less than or equal to n, wherein x i Is shown at t 0 Whether collection is carried out at + i × Δ t moment, 1 represents collection, 0 represents non-collection, and t represents collection 0 Δ t represents the minimum acquisition time interval for the current time; the frequency of collecting data is defined as:
the acquisition distortion factor is defined as: />
Wherein p is i Is represented at t 0 Predicted value at time + i Δ t, s i Representing that the corresponding time point t on the obtained fitting curve is assumed to acquire data according to the X 0 And (3) taking the frequency and the collection distortion degree of the collected data as optimization targets according to the data values on + i × Δ t, thereby obtaining a multi-target optimization problem: minG (X) = (F (X), R (X)); weighting the multi-objective optimization problem by a target weighting method to obtain a new objective function: g (X) = (1-w) r )F(X)+w r R (X), wherein w r As weight parameter, 0 < w r Less than 1; and solving the new objective function by using a genetic algorithm to obtain the optimal acquisition time sequence X corresponding to each equipment minG (X). />
As an alternative embodiment, the updating module 204 is specifically configured to obtain one optimal acquisition time sequence X for each acquired data device when the update threshold of the acquisition time schedule is reached;
replacing one binary number Y of n bits maintained for each device for which data is acquired in the acquisition schedule with the optimal acquisition time sequence, i.e., binary number X of n bits.
As an optional embodiment, the adjusting module 205 is specifically configured to set an idle threshold for each device with the collected data to be an idle period CPU utilization U free Setting busy threshold as CPU utilization rate U in busy time period busy (ii) a Polling the device periodically when the CPU utilization U of the device cpu Is less than the U free Reducing the weight parameter w in the new objective function r Increasing the frequency of the acquired data; when U of the device cpu Is greater than the U busy Increasing the weight parameter w in the new objective function r The frequency of the collected data is reduced.
For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, the functionality of the modules may be implemented in the same one or more software and/or hardware implementations in implementing one or more embodiments of the present description.
The apparatus of the foregoing embodiment is used to implement the adaptive network data acquisition method based on multi-objective optimization in the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described herein again.
Based on the same inventive concept, corresponding to any of the above embodiments, one or more embodiments of the present specification further provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the method for acquiring adaptive network data based on multi-objective optimization according to any of the above embodiments.
Fig. 3 is a schematic diagram illustrating a more specific hardware structure of an electronic device according to this embodiment, where the electronic device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein the processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 are communicatively coupled to each other within the device via a bus 1050.
The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present disclosure.
The Memory 1020 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 1020 may store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and called by the processor 1010 for execution.
The input/output interface 1030 is used for connecting an input/output module to input and output information. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The communication interface 1040 is used for connecting a communication module (not shown in the drawings) to implement communication interaction between the present apparatus and other apparatuses. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, bluetooth and the like).
It should be noted that although the above-mentioned device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present description, and not necessarily all of the components shown in the figures.
The electronic device of the above embodiment is used to implement the corresponding multi-objective optimization-based adaptive network data acquisition method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, one or more embodiments of the present specification further provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to execute the multi-objective optimization-based adaptive network data acquisition method according to any of the above-mentioned embodiments.
Computer-readable media of the present embodiments, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device.
The computer instructions stored in the storage medium of the foregoing embodiment are used to implement the adaptive network data acquisition method based on multi-objective optimization in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which are not described herein again.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the spirit of the present disclosure, features from the above embodiments or from different embodiments may also be combined, steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present description as described above, which are not provided in detail for the sake of brevity.
In addition, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown in the provided figures, for simplicity of illustration and discussion, and so as not to obscure one or more embodiments of the disclosure. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the understanding of one or more embodiments of the present description, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the one or more embodiments of the present description are to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that one or more embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures, such as Dynamic RAM (DRAM), may use the discussed embodiments.
It is intended that the one or more embodiments of the present specification embrace all such alternatives, modifications and variations as fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of one or more embodiments of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (7)
1. A multi-objective optimization-based adaptive network data acquisition method is characterized by comprising the following steps:
data acquisition is carried out on the equipment according to an acquisition time schedule, and the acquired data are stored in a database;
based on historical collected data in the database, calculating to obtain a stationarity value of the data collected by the equipment at the t time
The trend value->
And seasonal value>
；
By using the Holt-Winters method, the prediction process is carried out
Performance index data prediction value after minimum acquisition interval duration
The prediction formula is:
wherein, the first and the second end of the pipe are connected with each other,
can be selected by>
Calculating to obtain;
calculated from the above prediction formula
Sequence of predicted values for individual time pointsP，/>
，/>
Is represented as being->
A prediction value at a moment in time, wherein>
，/>
For the current time>
Is the minimum acquisition time interval;
setting the optimal acquisition time sequence of any one device obtained by solving through a genetic algorithm based on a target weighting method as
Binary number of bits>
，/>
，/>
In which>
Is indicated to be at>
Whether or not to collect at that moment, 1 represents collection, 0 represents no collection, and->
Is the current time->
Represents a minimum acquisition time interval;
the frequency of collecting data is defined as:
the acquisition distortion factor is defined as: />
In which>
Is represented as being->
The predicted value at that moment is greater or less>
Indicating that the hypothesis is according to>
Collecting data, and obtaining a corresponding time point on the fitting curve>
The data value is optimized by taking the frequency and the distortion degree of the collected data as optimization targets, so that a multi-target optimization problem is obtained: />
；
Weighting the multi-objective optimization problem by a target weighting method to obtain a new objective function:
wherein the content of the first and second substances,
is a weightThe parameters are set to be in a predetermined range,0＜/>
＜1；
solving the new objective function by genetic algorithm to obtain each of the devices
Corresponding optimal sequence of acquisition times->
；
When the updating threshold value of the acquisition time table is reached, calling the optimal acquisition time sequence X to update the acquisition time table;
and when the CPU utilization rate of the equipment is smaller than an idle threshold or larger than a busy threshold, adjusting the weight parameter in the new objective function to change the frequency of the acquired data.
2. The method of claim 1, wherein the collecting data of the device according to the collection schedule and storing the collected data in a database comprises:
one for each device in the acquisition schedule for which data is acquired
Binary number of bits>
，
，/>
，/>
Wherein is present>
Is indicated to be at>
Whether or not to collect at that moment, 1 represents collection, 0 represents no collection, and->
Is the current time->
Is the minimum acquisition time interval;
every other
Traversing the collection schedule, and->
If the number of the equipment is 1, adding the identification OID corresponding to the equipment into a list to be acquired, and if the number of the equipment is 0, not operating;
constructing a Simple Network Management Protocol (SNMP) request according to the list to be acquired, and initiating a data acquisition request to each device;
and storing the acquired data into a database.
3. The method according to claim 1, wherein said invoking the optimal acquisition time sequence to update the acquisition schedule when the update threshold of the acquisition schedule is reached comprises:
each of said devices of the acquired data gets one of said optimal acquisition time sequences when an update threshold of said acquisition schedule is reached
；
Using said optimal acquisition time sequence, i.e.
Binary number of bits->
Replacement of a ≧ maintained for each device whose data is collected in the collection schedule>
Binary number of bits->
。
4. The method according to claim 1, wherein when the CPU utilization of the device is less than an idle threshold or greater than a busy threshold, adjusting the weight parameter in the new objective function to change the frequency of data collection comprises:
setting an idle threshold for each acquired data device as an idle time period CPU utilization
Setting a busy threshold to be CPU utilization->
；
Polling said device periodically when CPU utilization of said device
Is less than or equal to>
Decreasing the weight parameter in the new objective function->
Increasing the frequency of the collected data;
when the device is in use
Greater than said +>
When the weight parameter ≥ in the new objective function is increased>
The frequency of the collected data is reduced.
5. An adaptive network data acquisition device based on multi-objective optimization, comprising:
the acquisition module is configured to acquire data of the equipment according to an acquisition time schedule and store the acquired data into a database;
a prediction module configured to
Based on historical collected data in the database, calculating to obtain a stationarity value of the data collected by the equipment at the t time
The trend value->
And a seasonal value->
；
By using the Holt-Winters method, the prediction process is carried out
Performance index data prediction value after minimum acquisition interval duration
The prediction formula is:
wherein the content of the first and second substances,
can be selected by>
Calculating to obtain;
calculated by the above prediction formula
Sequence of predicted values for a time pointP，
，/>
Is represented as being->
A prediction value at a moment, wherein>
，/>
Is the current time->
A minimum acquisition time interval;
a computing module configured to
Setting the optimal acquisition time sequence of any equipment obtained by solving the genetic algorithm based on the target weighting method as
Binary number of bits->
，/>
，/>
Wherein is present>
Is indicated to be at>
Whether or not to collect at that moment, 1 represents collection, 0 represents no collection, and->
Is the current time->
Represents a minimum acquisition time interval;
the frequency of collecting data is defined as:
the acquisition distortion factor is defined as:
wherein is present>
Is represented as being->
A predicted value at a moment>
Indicating that the hypothesis is according to>
Collecting data, and obtaining a corresponding time point on the fitting curve>
The data value of (2), the frequency of collecting data and the collection distortion factor are used as optimization purposesThe objective is to obtain a multi-objective optimization problem: />
；
Weighting the multi-objective optimization problem by a target weighting method to obtain a new objective function:
wherein the content of the first and second substances,
as a function of the weight parameter(s),0＜/>
＜1；
solving said new objective function using genetic algorithm to obtain each of said devices
Corresponding optimal sequence of acquisition times->
；；
An update module configured to invoke the optimal acquisition time sequence to update the acquisition schedule when an update threshold of the acquisition schedule is reached;
an adjusting module configured to adjust a weight parameter in the new objective function to change a frequency of acquiring data when a CPU utilization of the device is less than an idle threshold or greater than a busy threshold.
6. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 4 when executing the program.
7. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 4.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011223995.XA CN112307044A (en) | 2020-11-05 | 2020-11-05 | Adaptive network data acquisition method based on multi-objective optimization and related equipment |
| CN202011223995X | 2020-11-05 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112732738A CN112732738A (en) | 2021-04-30 |
| CN112732738B true CN112732738B (en) | 2023-04-18 |
Family
ID=74326287
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011223995.XA Withdrawn CN112307044A (en) | 2020-11-05 | 2020-11-05 | Adaptive network data acquisition method based on multi-objective optimization and related equipment |
| CN202110210085.6A Active CN112732738B (en) | 2020-11-05 | 2021-02-24 | Adaptive network data acquisition method based on multi-objective optimization and related equipment |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011223995.XA Withdrawn CN112307044A (en) | 2020-11-05 | 2020-11-05 | Adaptive network data acquisition method based on multi-objective optimization and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN112307044A (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112953792B (en) * | 2021-02-05 | 2023-09-19 | 中国工商银行股份有限公司 | Network traffic monitoring method and device |
| CN113684868B (en) * | 2021-08-31 | 2023-03-21 | 中煤科工集团重庆智慧城市科技研究院有限公司 | City utility tunnel early warning system |
| CN114217017A (en) * | 2021-12-07 | 2022-03-22 | 杭州中科先进技术研究院有限公司 | Deep learning-based energy-saving interactive negative oxygen ion detection method and system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104065168B (en) * | 2014-06-30 | 2016-01-20 | 江苏华大天益电力科技有限公司 | A kind of dynamic frequency-conversion collecting method of wind-solar-storage joint generating state monitoring |
| US10049301B2 (en) * | 2016-08-01 | 2018-08-14 | Siemens Healthcare Gmbh | Medical scanner teaches itself to optimize clinical protocols and image acquisition |
| CN108173670B (en) * | 2016-12-07 | 2020-06-02 | 华为技术有限公司 | Method and device for detecting network |
| CN107070254B (en) * | 2017-04-13 | 2019-03-19 | 湖南科技大学 | A kind of Buck-Boost matrix converter parameter optimization method and device |
-
2020
- 2020-11-05 CN CN202011223995.XA patent/CN112307044A/en not_active Withdrawn
-
2021
- 2021-02-24 CN CN202110210085.6A patent/CN112732738B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112732738A (en) | 2021-04-30 |
| CN112307044A (en) | 2021-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112732738B (en) | Adaptive network data acquisition method based on multi-objective optimization and related equipment | |
| CN110852421B (en) | Model generation method and device | |
| JP2022003566A (en) | Correlation between thread strength and heep use amount for specifying stack trace in which heaps are stored up | |
| CN108763398B (en) | Database configuration parameter processing method and device, computer equipment and storage medium | |
| CN106454437B (en) | A kind of streaming media service rate prediction method and device | |
| WO2021129086A1 (en) | Traffic prediction method, device, and storage medium | |
| CN111914516B (en) | Method, device, equipment and storage medium for generating network data prediction sequence | |
| CN112149797B (en) | Neural network structure optimization method and device and electronic equipment | |
| CN111371626B (en) | Bandwidth prediction method based on neural network | |
| CN112579194A (en) | Block chain consensus task unloading method and device based on time delay and transaction throughput | |
| CN112882889A (en) | Abnormality monitoring method, abnormality monitoring system, electronic device, and storage medium | |
| CN114500339B (en) | Node bandwidth monitoring method and device, electronic equipment and storage medium | |
| US20220335297A1 (en) | Anticipatory Learning Method and System Oriented Towards Short-Term Time Series Prediction | |
| WO2023029680A1 (en) | Method and apparatus for determining usable duration of magnetic disk | |
| CN111949498A (en) | Application server abnormity prediction method and system | |
| CN112131274B (en) | Method, device, equipment and readable storage medium for detecting abnormal points of time sequence | |
| CN111813524A (en) | Task execution method and device, electronic equipment and storage medium | |
| CN109978172B (en) | Resource pool utilization rate prediction method and device based on extreme learning machine | |
| CN114157578B (en) | Network state prediction method and device | |
| CN116308450A (en) | Information generation and model training method, electronic equipment and storage medium | |
| CN112989203B (en) | Material throwing method, device, equipment and medium | |
| US20220188383A1 (en) | Horizon-based smoothing of forecasting model | |
| CN111666535B (en) | Method and device for determining active time length of user, electronic equipment and storage medium | |
| WO2016165471A1 (en) | Method and apparatus for collecting index of hadoop trunking component | |
| CN113420165A (en) | Training of two-classification model and classification method and device of multimedia data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |