CN112732738A - Adaptive network data acquisition method based on multi-objective optimization and related equipment - Google Patents
Adaptive network data acquisition method based on multi-objective optimization and related equipment Download PDFInfo
- Publication number
- CN112732738A CN112732738A CN202110210085.6A CN202110210085A CN112732738A CN 112732738 A CN112732738 A CN 112732738A CN 202110210085 A CN202110210085 A CN 202110210085A CN 112732738 A CN112732738 A CN 112732738A
- Authority
- CN
- China
- Prior art keywords
- acquisition
- data
- collection
- objective function
- acquired
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/12—Computing arrangements based on biological models using genetic models
- G06N3/126—Evolutionary algorithms, e.g. genetic algorithms or genetic programming
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Genetics & Genomics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Physiology (AREA)
- Databases & Information Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the specification provides a self-adaptive network data acquisition method based on multi-objective optimization and related equipment; the method comprises the following steps: predicting data to be collected by utilizing a Holt-Winters method; quantizing the frequency and the acquisition distortion of the acquired data, and constructing a multi-objective optimization problem; converting the multi-objective optimization problem into a new objective function through an objective weighting method, and solving the new objective function by using a genetic algorithm based on prediction data to obtain an optimal acquisition time sequence; and dynamically adjusting the weight parameters in the new objective function according to the CPU utilization rate of the adopted equipment. Compared with the traditional periodic acquisition method, the method provided by the specification considers the change of the acquired data, reduces the frequency of the acquired data by reasonably distributing acquisition time points, reduces the distortion caused by the acquisition process as much as possible, takes the CPU utilization rate of the acquired equipment into consideration range, and avoids causing overlarge acquisition burden.
Description
Technical Field
One or more embodiments of the present disclosure relate to the field of digital information transmission and data acquisition technologies, and in particular, to an adaptive network data acquisition method and related device based on multi-objective optimization.
Background
In the current internet application scenario, data becomes more and more important, and the data is a basis for supporting realization of many services, for example, by collecting network data and further analyzing characteristics of the data to detect attacks and intrusions, so as to realize protection of a network system, but at present, data collection is a performance bottleneck of most service systems related to the data.
The data acquisition algorithm with equal time intervals adopted in a large amount of existing network management software periodically acquires data of managed equipment according to acquisition frequency set by a system or a user, and changes of the acquired data are ignored. For example, when the acquired data changes violently, the change trend of the acquired object cannot be accurately reflected due to too low acquisition frequency, and important data is easily lost; when the acquired data is basically unchanged, if the acquisition frequency is too high, data redundancy and network resource waste are caused, and meanwhile, because the requested data volume is large, a Simple Network Management Protocol (SNMP) agent of the equipment can not respond to the request, even occupies too high a Central Processing Unit (CPU), and other normal services of the network equipment are influenced.
Based on this, a method capable of adaptively acquiring network data is needed.
Disclosure of Invention
In view of this, an object of one or more embodiments of the present disclosure is to provide an adaptive network data acquisition method and related device based on multi-objective optimization.
In view of the above, one or more embodiments of the present specification provide an adaptive network data acquisition method based on multi-objective optimization, including:
data acquisition is carried out on the equipment according to an acquisition time schedule, and the acquired data are stored in a database;
based on historical collected data in the database, calculating a predicted value of the collected data in the next time period by using a Holt-Winters method;
based on the predicted value, establishing a multi-objective optimization problem by taking the frequency and the collection distortion of collected data as optimization targets, then obtaining a new objective function by a target weighting method, and solving the new objective function by using a genetic algorithm to obtain an optimal collection time sequence;
when the updating threshold value of the acquisition time table is reached, calling the optimal acquisition time sequence to update the acquisition time table;
and when the CPU utilization rate of the equipment is smaller than an idle threshold or larger than a busy threshold, adjusting the weight parameter in the new objective function to change the frequency of the acquired data.
Based on the same inventive concept, one or more embodiments of the present specification further provide an adaptive network data acquisition apparatus based on multi-objective optimization, including:
the acquisition module is configured to acquire data of the equipment according to an acquisition time schedule and store the acquired data into a database;
the prediction module is configured to calculate a predicted value of the acquired data in the next time period by utilizing a Holt-Winters method based on historical acquired data in the database;
the calculation module is configured to construct a multi-objective optimization problem by taking the frequency of acquired data and the acquisition distortion as optimization targets based on the predicted values, then obtain a new objective function through a target weighting method, and solve the new objective function by using a genetic algorithm to obtain an optimal acquisition time sequence;
an update module configured to invoke the optimal acquisition time sequence to update the acquisition schedule when an update threshold of the acquisition schedule is reached;
an adjusting module configured to adjust a weight parameter in the new objective function to change a frequency of acquiring data when a CPU utilization of the device is less than an idle threshold or greater than a busy threshold.
Based on the same inventive concept, one or more embodiments of the present specification further provide an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the method as described in any one of the above items when executing the program.
Based on the same inventive concept, one or more embodiments of the present specification also provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in any one of the above.
As can be seen from the foregoing, in the adaptive network data acquisition method and the related device based on multi-objective optimization provided in one or more embodiments of the present specification, changes of acquired data are considered, and a genetic algorithm based on target weighting is used to minimize the frequency of acquired data and the acquisition distortion, so that the frequency of acquired data is reduced, distortion caused by an acquisition process is reduced as much as possible, the accuracy of acquired data is improved, and meanwhile, the CPU utilization of the acquired device is taken into consideration, thereby avoiding an extra performance burden of the device due to acquisition.
Drawings
In order to more clearly illustrate one or more embodiments or prior art solutions of the present specification, the drawings that are needed in the description of the embodiments or prior art will be briefly described below, and it is obvious that the drawings in the following description are only one or more embodiments of the present specification, and that other drawings may be obtained by those skilled in the art without inventive effort from these drawings.
FIG. 1 is a flow diagram of a method for adaptive network data collection based on multi-objective optimization according to one or more embodiments of the present disclosure;
FIG. 2 is a schematic structural diagram of an adaptive network data acquisition device based on multi-objective optimization according to one or more embodiments of the present disclosure;
fig. 3 is a schematic structural diagram of an electronic device according to one or more embodiments of the present disclosure.
Detailed Description
For the purpose of promoting a better understanding of the objects, aspects and advantages of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.
It is to be noted that unless otherwise defined, technical or scientific terms used in one or more embodiments of the present specification should have the ordinary meaning as understood by those of ordinary skill in the art to which this disclosure belongs. The use of the terms "comprising" or "including" and the like in one or more embodiments of the present specification is intended to mean that the element or item presented before the term "comprises" or "comprising" is included in the list of elements or items listed after the term and its equivalents, without excluding other elements or items.
As described in the background section, the existing network data acquisition method ignores the change of the acquired data, thereby causing the problems of large acquisition distortion, inaccurate acquired data, increased equipment acquisition burden and the like. At present, some techniques propose adaptive acquisition to dynamically adjust the frequency of acquired data, which dynamically adjusts the frequency of acquired data according to the changing situation of the acquired object.
The applicant finds that the existing adaptive network data acquisition method has some defects in the process of implementing the disclosure: the acquisition frequency is adjusted according to the recent data fluctuation degree, the change of an acquisition object is calculated in real time, and the calculation consumption is large and the accuracy is insufficient; adjustment of the acquisition frequency is usually performed by setting an adjustment rule according to the data fluctuation degree by subjective experience, so that the adjustment scale is difficult to accurately grasp, the acquisition efficiency or the acquisition precision is easy to reduce, and the reliability is poor; the influence of the collection behavior on the collected equipment is not considered, so that the collected equipment is frequently collected when the pressure of the collected equipment is high, and serious consequences such as equipment downtime and the like are caused.
In view of this, one or more embodiments of the present disclosure provide an adaptive network data acquisition method based on multi-objective optimization, and specifically, data acquisition is performed on devices according to an acquisition schedule, and the acquired data is stored in a database, where the acquisition schedule stores the acquisition data time of each acquired device. And then, based on the historical collected data in the database, calculating by using a Holt-Winters method to obtain a predicted value of the collected data in the next time period. In addition, based on the predicted value, the acquisition strategy target is quantized into two optimization targets, namely, the frequency and the acquisition distortion of acquired data are used as optimization targets, a multi-target optimization problem is constructed, then a new target function is obtained through a target weighting method, and the new target function is solved through a genetic algorithm to obtain an optimal acquisition time sequence. Further, when the update threshold of the acquisition schedule is reached, the optimal acquisition time sequence is called to update the acquisition schedule. And finally, using the CPU utilization rate of the collected equipment as a measurement standard, and dynamically adjusting the weight parameter in the new objective function to change the frequency of the collected data when the CPU utilization rate of the equipment is smaller than an idle threshold or larger than a busy threshold.
Therefore, the adaptive network data acquisition method based on multi-objective optimization in one or more embodiments of the present specification considers the change of the acquired data, reduces the frequency of the acquired data, reduces the acquisition distortion as much as possible, improves the accuracy of the acquired data, and simultaneously takes the CPU utilization of the acquired device into consideration, thereby avoiding the extra and overlarge performance burden of the device due to acquisition.
The technical solutions of one or more embodiments of the present specification are described in detail below with reference to specific embodiments.
Referring to fig. 1, an adaptive network data acquisition method based on multi-objective optimization in one embodiment of the present specification includes the following steps:
and S101, acquiring data of the equipment according to an acquisition time schedule, and storing the acquired data into a database.
In this step, an n-bit binary Y ═ Y is maintained for each device to be acquired in the acquisition schedule1,y2,…,yn-1,yn}，yiE is {0,1}, i is more than or equal to 1 and less than or equal to n, wherein yi(1. ltoreq. i. ltoreq.n) at t0Whether collection is carried out at + i × Δ t moment, 1 represents collection, 0 represents non-collection, and t represents collection0As the current time, Δ t is the minimum acquisition time interval;
the step S101 specifically includes: traverse the acquisition schedule, y, every Δ tiIf the number of the equipment is 1, adding the identification OID corresponding to the equipment into a list to be acquired, and if the number of the equipment is 0, not operating;
after traversing, constructing a Simple Network Management Protocol (SNMP) request according to the list to be acquired, and initiating a data acquisition request to each device;
and storing the acquired data into a database.
And S102, calculating a predicted value of the acquired data in the next time period by utilizing a Holt-Winters method based on the historical acquired data in the database.
The step S102 specifically includes:
an initial value part that selects, based on the historical collected data in the database, the historical data of the first two cycles using the first predicted point of the data to be collected next for calculation:
the initial value of the stable value is:
the initial value of the trend value is:
the initial seasonal value is:
wherein Si' for each moment of time during the first cycle:
Si"is the seasonal value at each time in the second period:
wherein L is the cycle length, i.e. the number of acquisitions in each cycle, xiHistorical data values of the first two periods of the first predicted point of the next data to be collected;
obtaining the stability of the data acquired by the equipment at the t time by recursion according to the calculation formula of the initial value partProperty value atTrend value btAnd a seasonal value StWherein said at、bt、StThe calculation formula of (2) is as follows:
bt＝β(at-at-1)+(1-β)bt-1
a abovet、bt、StIn the calculation formula (2), the coefficients α, β and γ are independent variables which determine the predicted values of the performance index data corresponding to the respective stationarity, tendency and seasonality, and the value ranges thereof are all intervals (0, 1). Under the condition of sufficient historical data, corresponding predictions can be made in advance through any combination of 3 exhaustive smoothing coefficients (from 0.01 to 0.99), the sum of squares of relative error values is obtained through historical record calculation, and the smoothing coefficient corresponding to the minimum error is selected as a basic value of the optimal smoothing coefficient. During real-time prediction calculation, the collected data of the latest period is used as a predicted value, adjustment is carried out on a basic value, and alpha, beta and gamma with the minimum corresponding errors are selected;
predicting the performance index data predicted value x after h minimum acquisition interval durations by using Holt-Winters methodt+hThe prediction formula is:
xt+h＝(at+h*bt)St+h-L
wherein S ist+h-LCan be composed oftCalculated as h 1, 2, 3 … n, and when h is 1, xt+1Is the predicted value of the first time point to be acquired after 1 minimum acquisition interval duration, when h is 2, xt+2After 2 minimum acquisition interval durations, acquiring a predicted value of a second time point, and then successively and repeatedly calculating to obtain the next n time pointsP, P ═ P1,p2,…,pn-1,pn}，piIs represented at t0The predicted value at the moment of + i Δ t, wherein i is more than or equal to 1 and less than or equal to n, and t0At the current time, Δ t is the minimum acquisition time interval.
And S103, based on the predicted value, constructing a multi-objective optimization problem by taking the frequency and the collection distortion of collected data as optimization targets, then obtaining a new objective function by a target weighting method, and solving the new objective function by using a genetic algorithm to obtain an optimal collection time sequence.
The step S103 specifically includes: setting the best acquisition time sequence of any equipment solved by a genetic algorithm based on a target weighting method as a binary number X with n bits, wherein X is { X ═ X }1,x2,…,xn-1,xn}，xiIs belonged to {0,1}, i is more than or equal to 1 and less than or equal to n, wherein xiIs shown at t0Whether collection is carried out at + i × Δ t moment, 1 represents collection, 0 represents non-collection, and t represents collection0Δ t represents the minimum acquisition time interval for the current time;
the frequency of collecting data is defined as:
the acquisition distortion factor is defined as:
wherein p isiIs represented at t0Predicted value at time + i Δ t, siRepresenting the corresponding time point t on the fitting curve obtained by assuming that data are collected according to the X0And (3) taking the frequency and the collection distortion degree of the collected data as optimization targets according to the data values on + i × Δ t, thereby obtaining a multi-target optimization problem: ming (f), (x), r (x);
then, solving the multi-objective optimization problem by using a genetic algorithm based on an objective weighting method, firstly weighting the multi-objective optimization problem by using the objective weighting method, namely, dividing each optimized objective vector by a weight to obtain a new objective function:
G(X)＝(1-wr)F(X)+wrR(X)
wherein, wrAs weight parameter, 0 < wr＜1；
Then, solving the new objective function by using a genetic algorithm to obtain an optimal acquisition time sequence X corresponding to each device minG (X).
The step of solving the new objective function by using the genetic algorithm comprises the following steps:
(a) and initializing to generate an initial population. Since the solved binary number is the binary number, the encoding operation is not needed, and the binary number with the length of n bits is the individual in the population. Firstly, initializing a population Q with a population scale of Q, namely randomly generating Q binary numbers with n bit lengths;
(b) individual evaluation fitness of individuals in the population Q was evaluated for each individual gene using an adaptive function, i.e., the inverse number of g (x) described above, i.e., Q (x) ═ 1-wr)F(X)-wrR (X), and judging whether the condition K of the optimization criterion is met, namely Q (X) is greater than a preset target value Qmax. If the data are matched, outputting the best individual and stopping; if the current evolution times reach the maximum times EmaxOutputting the individual with the highest adaptability in the population; otherwise, continuing the next step;
(c) according to the selection probability, executing a selection operator, and selecting part of individuals from the current population to enter the next generation population;
(d) performing crossover operators according to crossover probabilities, wherein uniform crossover is used in the embodiment, and genes at each locus of two paired individuals are exchanged with the same crossover probability;
(e) executing mutation operators according to the mutation probability, wherein uniform mutation is used in the embodiment, and original gene values of all loci in the individual code strings are replaced by a certain smaller probability;
(f) and (c) generating a new generation of population by crossing and mutation, and returning to the step (b).
From this, the update value for each device, i.e. the optimal acquisition time series X, is found.
And step S104, calling the optimal acquisition time sequence to update the acquisition time table when the update threshold of the acquisition time table is reached.
The step S104 specifically includes: maintaining an n-bit binary number Y for each device to be acquired in the acquisition time table, reading a bit binary number every delta t, and when a w (1< w < n) th bit is read, reaching an update threshold of the acquisition time table, wherein each device to be acquired obtains the optimal acquisition time sequence X;
replacing an n-bit binary number Y maintained for each device of the acquired data in the acquisition time table with the optimal acquisition time sequence, i.e., the n-bit binary number X;
and acquiring data of the equipment according to the updated acquisition time table.
And S105, when the CPU utilization rate of the equipment is smaller than an idle threshold or larger than a busy threshold, adjusting the weight parameter in the new objective function to change the frequency of the acquired data.
The step S105 specifically includes: setting an idle threshold value for each acquired data device as an idle time period CPU utilization rate UfreeSetting busy threshold as CPU utilization rate U in busy time periodbusy；
Polling the device periodically when the CPU utilization U of the devicecpuIs less than the UfreeReducing the weight parameter w in the new objective functionrLet w ber＝wr(ii)/2, such that the frequency of said data acquisition is increased;
when U of the devicecpuGreater than or equal to UfreeAnd is less than or equal to UbusyThe weight parameter w in the new target function numberrThe change is not changed;
when U of the devicecpuIs greater than the UbusyIncreasing the weight parameter w in the new objective functionrLet w ber＝wr+ Δ w, such that the frequency of the acquired data is reduced.
It can be seen that the adaptive network data acquisition method based on multi-objective optimization provided in the embodiments of the present specification quantifies the frequency and the acquisition distortion of the acquired data, constructs a multi-objective optimization problem with the frequency and the acquisition distortion of the acquired data as optimization targets, solves the multi-objective optimization problem by using a genetic algorithm based on target weighting, and calculates an optimal acquisition time sequence in a future time period. Compared with a method for calculating an acquisition frequency, the method provided by the specification avoids frequent calculation and is more flexible, and meanwhile, binary storage is adopted, so that excessive storage resources are avoided being occupied; compared with the traditional periodic acquisition method, the method provided by the specification considers the change of the acquired data, reduces the frequency of the acquired data and the acquisition distortion as much as possible, predicts the value of the acquired data in a future time period by using a Holt-Winters method, calculates the optimal acquisition time point based on the value, improves the accuracy of the acquired data, takes the CPU utilization rate of the acquired equipment into consideration range, and avoids extra and overlarge performance burden of the equipment caused by acquisition.
It should be noted that the method of one or more embodiments of the present disclosure may be performed by a single device, such as a computer or server. The method of the embodiment can also be applied to a distributed scene and completed by the mutual cooperation of a plurality of devices. In such a distributed scenario, one of the devices may perform only one or more steps of the method of one or more embodiments of the present disclosure, and the devices may interact with each other to complete the method.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Based on the same inventive concept, corresponding to the method of any embodiment, one or more embodiments of the present specification further provide an adaptive network data acquisition device based on multi-objective optimization. Referring to fig. 2, the adaptive network data acquisition device based on multi-objective optimization includes:
the acquisition module 201 is configured to acquire data of the equipment according to an acquisition schedule and store the acquired data in a database;
the prediction module 202 is configured to calculate a predicted value of the acquired data in the next time period by using a Holt-Winters method based on historical acquired data in the database;
the calculation module 203 is configured to construct a multi-objective optimization problem by taking the frequency of the acquired data and the acquisition distortion as optimization targets based on the predicted values, obtain a new objective function by a target weighting method, and solve the new objective function by using a genetic algorithm to obtain an optimal acquisition time sequence;
an update module 204 configured to invoke the optimal acquisition time sequence to update the acquisition schedule when an update threshold of the acquisition schedule is reached;
an adjusting module 205 configured to adjust weight parameters in the new objective function to change a frequency of acquiring data when a CPU utilization of the device is less than an idle threshold or greater than a busy threshold.
As an alternative embodiment, the collecting module 201 is specifically configured to maintain an n-bit binary number Y, Y ═ Y for each device to be collected in the collection schedule1,y2,…,yn-1,yn}，yiE is {0,1}, i is more than or equal to 1 and less than or equal to n, wherein yiIs shown at t0Whether collection is carried out at + i × Δ t moment, 1 represents collection, 0 represents non-collection, and t represents collection0As the current time, Δ t is the minimum acquisition time interval; traverse the acquisition schedule, y, every Δ tiIf the number of the equipment is 1, adding the identification OID corresponding to the equipment into a list to be acquired, and if the number of the equipment is 0, not operating; constructing a Simple Network Management Protocol (SNMP) request according to the list to be acquired, and initiating a data acquisition request to each device; and storing the acquired data into a database.
As an alternative embodiment, the prediction module 202 is specifically configured to calculate a stationarity value a of the data acquired by the device at the t-th time based on the historical acquired data in the databasetTrend value btAnd a seasonal value St(ii) a Predicting the performance index data predicted value x after h minimum acquisition interval durations by using Holt-Winters methodt+hThe prediction formula is: x is the number oft+h＝(at+h*bt)St+h-LWherein S ist+h-LCan be composed oftCalculating to obtain; calculating a predicted value sequence P of the next n time points by the prediction formula, wherein P is { P ═ P1,p2,…,pn-1,pn}，piIs represented at t0The predicted value at the moment of + i Δ t, wherein i is more than or equal to 1 and less than or equal to n, and t0At the current time, Δ t is the minimum acquisition time interval.
As an alternative embodiment, the calculating module 203 is specifically configured to set the optimal acquisition time sequence of any device solved by the genetic algorithm based on the target weighting method to be a binary number X with n bits, where X is { X ═ X {1,x2,…,xn-1,xn}，xiE is {0,1}, i is more than or equal to 1 and less than or equal to n, wherein xiIs shown at t0Whether collection is carried out at + i × Δ t moment, 1 represents collection, 0 represents non-collection, and t represents collection0Δ t represents the minimum acquisition time interval for the current time; the frequency of collecting data is defined as:
the acquisition distortion factor is defined as:
wherein p isiIs represented at t0Predicted value at time + i Δ t, siRepresenting the corresponding time point t on the fitting curve obtained by assuming that data are collected according to the X0And (3) taking the frequency and the collection distortion degree of the collected data as optimization targets according to the data values on + i × Δ t, thereby obtaining a multi-target optimization problem: ming (f), (x), r (x); weighting the multi-objective optimization problem by a target weighting method to obtain a new objective function: g (x) ═ 1-wr)F(X)+wrR (X), wherein wrAs weight parameter, 0 < wrLess than 1; and solving the new objective function by using a genetic algorithm to obtain the optimal acquisition time sequence X corresponding to each equipment minG (X).
As an alternative embodiment, the updating module 204 is specifically configured to obtain one optimal acquisition time sequence X for each acquired data device when the update threshold of the acquisition time schedule is reached;
the best acquisition time sequence, n-bit binary number X, is used to replace one n-bit binary number Y maintained for each device whose data is acquired in the acquisition schedule.
As an optional embodiment, the adjusting module 205 is specifically configured to set an idle threshold for each device with the collected data to be an idle period CPU utilization UfreeSetting busy threshold as CPU utilization rate U in busy time periodbusy(ii) a Polling the device periodically when the CPU utilization U of the devicecpuIs less than the UfreeReducing the weight parameter w in the new objective functionrIncreasing the frequency of the collected data; when U of the devicecpuIs greater than the UbusyIncreasing the weight parameter w in the new objective functionrThe frequency of the collected data is reduced.
For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, the functionality of the modules may be implemented in the same one or more software and/or hardware implementations in implementing one or more embodiments of the present description.
The apparatus of the foregoing embodiment is used to implement the corresponding adaptive network data acquisition method based on multi-objective optimization in the foregoing embodiment, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Based on the same inventive concept, corresponding to any of the above embodiments, one or more embodiments of the present specification further provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the method for acquiring adaptive network data based on multi-objective optimization according to any of the above embodiments.
Fig. 3 is a schematic diagram illustrating a more specific hardware structure of an electronic device according to this embodiment, where the electronic device may include: a processor 1010, a memory 1020, an input/output interface 1030, a communication interface 1040, and a bus 1050. Wherein the processor 1010, memory 1020, input/output interface 1030, and communication interface 1040 are communicatively coupled to each other within the device via bus 1050.
The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the present disclosure.
The Memory 1020 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 1020 may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present specification is implemented by software or firmware, the relevant program codes are stored in the memory 1020 and called to be executed by the processor 1010.
The input/output interface 1030 is used for connecting an input/output module to input and output information. The i/o module may be configured as a component in a device (not shown) or may be external to the device to provide a corresponding function. The input devices may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output devices may include a display, a speaker, a vibrator, an indicator light, etc.
The communication interface 1040 is used for connecting a communication module (not shown in the drawings) to implement communication interaction between the present apparatus and other apparatuses. The communication module can realize communication in a wired mode (such as USB, network cable and the like) and also can realize communication in a wireless mode (such as mobile network, WIFI, Bluetooth and the like).
It should be noted that although the above-mentioned device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will appreciate that the above-described apparatus may also include only those components necessary to implement the embodiments of the present description, and not necessarily all of the components shown in the figures.
The electronic device of the above embodiment is used to implement the corresponding multi-objective optimization-based adaptive network data acquisition method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which are not described herein again.
Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, one or more embodiments of the present specification further provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to execute the multi-objective optimization-based adaptive network data acquisition method according to any of the above-mentioned embodiments.
Computer-readable media of the present embodiments, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
The computer instructions stored in the storage medium of the foregoing embodiment are used to implement the adaptive network data acquisition method based on multi-objective optimization in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which are not described herein again.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, is limited to these examples; within the spirit of the present disclosure, features from the above embodiments or from different embodiments may also be combined, steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present description as described above, which are not provided in detail for the sake of brevity.
In addition, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown in the provided figures, for simplicity of illustration and discussion, and so as not to obscure one or more embodiments of the disclosure. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the understanding of one or more embodiments of the present description, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform within which the one or more embodiments of the present description are to be implemented (i.e., specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the disclosure, it should be apparent to one skilled in the art that one or more embodiments of the disclosure can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative instead of restrictive.
While the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic ram (dram)) may use the discussed embodiments.
It is intended that the one or more embodiments of the present specification embrace all such alternatives, modifications and variations as fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of one or more embodiments of the present disclosure are intended to be included within the scope of the present disclosure.
Claims (9)
1. A multi-objective optimization-based adaptive network data acquisition method is characterized by comprising the following steps:
data acquisition is carried out on the equipment according to an acquisition time schedule, and the acquired data are stored in a database;
based on historical collected data in the database, calculating a predicted value of the collected data in the next time period by using a Holt-Winters method;
based on the predicted value, establishing a multi-objective optimization problem by taking the frequency and the collection distortion of collected data as optimization targets, then obtaining a new objective function by a target weighting method, and solving the new objective function by using a genetic algorithm to obtain an optimal collection time sequence;
when the updating threshold value of the acquisition time table is reached, calling the optimal acquisition time sequence to update the acquisition time table;
and when the CPU utilization rate of the equipment is smaller than an idle threshold or larger than a busy threshold, adjusting the weight parameter in the new objective function to change the frequency of the acquired data.
2. The method of claim 1, wherein the collecting data of the device according to the collection schedule and storing the collected data in a database comprises:
maintaining an n-bit binary number Y for each device in the collection schedule for the data being collected, Y ═ Y1,y2,…,yn-1,yn}，yiE is {0,1}, i is more than or equal to 1 and less than or equal to n, wherein yiIs shown at t0Whether collection is carried out at + i × Δ t moment, 1 represents collection, 0 represents non-collection, and t represents collection0As the current time, Δ t is the minimum acquisition time interval;
traverse the acquisition schedule, y, every Δ tiIf the number of the equipment is 1, adding the identification OID corresponding to the equipment into a list to be acquired, and if the number of the equipment is 0, not operating;
constructing a Simple Network Management Protocol (SNMP) request according to the list to be acquired, and initiating a data acquisition request to each device;
and storing the acquired data into a database.
3. The method according to claim 1, wherein the step of calculating a predicted value of the collected data in the next time period by using a Holt-Winters method based on the historical collected data in the database specifically comprises:
based on historical collected data in the database, calculating to obtain a stationarity value a of the data collected by the equipment at the t timetTrend value btAnd a seasonal value St；
Predicting the performance index data predicted value x after h minimum acquisition interval durations by using Holt-Winters methodt+hThe prediction formula is:
xt+h＝(at+h*bt)St+h-L
wherein S ist+h-LCan be composed oftCalculating to obtain;
calculating the next n times by the above prediction formulaPredicted value sequence P of points, P ═ P1,p2,…,pn-1,pn}，piIs represented at t0The predicted value at the moment of + i Δ t, wherein i is more than or equal to 1 and less than or equal to n, and t0At the current time, Δ t is the minimum acquisition time interval.
4. The method according to claim 3, wherein the constructing a multi-objective optimization problem based on the predicted values and using the frequency and the collection distortion of the collected data as optimization objectives, then obtaining a new objective function by an objective weighting method, and solving the new objective function by using a genetic algorithm to obtain an optimal collection time sequence specifically comprises:
setting the best acquisition time sequence of any equipment solved by a genetic algorithm based on a target weighting method as a binary number X with n bits, wherein X is { X ═ X }1,x2,…,xn-1,xn}，xiE is {0,1}, i is more than or equal to 1 and less than or equal to n, wherein xiIs shown at t0Whether collection is carried out at + i × Δ t moment, 1 represents collection, 0 represents non-collection, and t represents collection0Δ t represents the minimum acquisition time interval for the current time;
the frequency of collecting data is defined as:
the acquisition distortion factor is defined as:
wherein p isiIs represented at t0Predicted value at time + i Δ t, siRepresenting the corresponding time point t on the fitting curve obtained by assuming that data are collected according to the X0And (3) taking the frequency and the collection distortion degree of the collected data as optimization targets according to the data values on + i × Δ t, thereby obtaining a multi-target optimization problem: ming (f), (x), r (x);
weighting the multi-objective optimization problem by a target weighting method to obtain a new objective function:
G(X)＝(1-wr)F(X)+wrR(X)
wherein, wrAs weight parameter, 0 < wr＜1；
And solving the new objective function by using a genetic algorithm to obtain the optimal acquisition time sequence X corresponding to each equipment minG (X).
5. The method according to claim 4, wherein the invoking the optimal acquisition time sequence to update the acquisition schedule when the update threshold of the acquisition schedule is reached comprises:
when the update threshold of the acquisition schedule is reached, the each acquired data device obtains the optimal acquisition time sequence X;
the best acquisition time sequence, n-bit binary number X, is used to replace one n-bit binary number Y maintained for each device whose data is acquired in the acquisition schedule.
6. The method according to claim 4, wherein when the CPU utilization of the device is less than an idle threshold or greater than a busy threshold, adjusting the weight parameter in the new objective function to change the frequency of the collected data comprises:
setting an idle threshold value for each acquired data device as an idle time period CPU utilization rate UfreeSetting busy threshold as CPU utilization rate U in busy time periodbusy；
Polling the device periodically when the CPU utilization U of the devicecpuIs less than the UfreeReducing the weight parameter w in the new objective functionrIncreasing the frequency of the collected data;
when U of the devicecpuIs greater than the UbusyIncreasing the weight parameter w in the new objective functionrThe frequency of the collected data is reduced.
7. An adaptive network data acquisition device based on multi-objective optimization, comprising:
the acquisition module is configured to acquire data of the equipment according to an acquisition time schedule and store the acquired data into a database;
the prediction module is configured to calculate a predicted value of the acquired data in the next time period by utilizing a Holt-Winters method based on historical acquired data in the database;
the calculation module is configured to construct a multi-objective optimization problem by taking the frequency of acquired data and the acquisition distortion as optimization targets based on the predicted values, then obtain a new objective function through a target weighting method, and solve the new objective function by using a genetic algorithm to obtain an optimal acquisition time sequence;
an update module configured to invoke the optimal acquisition time sequence to update the acquisition schedule when an update threshold of the acquisition schedule is reached;
an adjusting module configured to adjust a weight parameter in the new objective function to change a frequency of acquiring data when a CPU utilization of the device is less than an idle threshold or greater than a busy threshold.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 6 when executing the program.
9. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 6.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011223995X | 2020-11-05 | ||
| CN202011223995.XA CN112307044A (en) | 2020-11-05 | 2020-11-05 | Adaptive network data acquisition method based on multi-objective optimization and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112732738A true CN112732738A (en) | 2021-04-30 |
| CN112732738B CN112732738B (en) | 2023-04-18 |
Family
ID=74326287
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011223995.XA Withdrawn CN112307044A (en) | 2020-11-05 | 2020-11-05 | Adaptive network data acquisition method based on multi-objective optimization and related equipment |
| CN202110210085.6A Active CN112732738B (en) | 2020-11-05 | 2021-02-24 | Adaptive network data acquisition method based on multi-objective optimization and related equipment |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011223995.XA Withdrawn CN112307044A (en) | 2020-11-05 | 2020-11-05 | Adaptive network data acquisition method based on multi-objective optimization and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN112307044A (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112953792B (en) * | 2021-02-05 | 2023-09-19 | 中国工商银行股份有限公司 | Network traffic monitoring method and device |
| CN113684868B (en) * | 2021-08-31 | 2023-03-21 | 中煤科工集团重庆智慧城市科技研究院有限公司 | City utility tunnel early warning system |
| CN114217017A (en) * | 2021-12-07 | 2022-03-22 | 杭州中科先进技术研究院有限公司 | Deep learning-based energy-saving interactive negative oxygen ion detection method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104065168A (en) * | 2014-06-30 | 2014-09-24 | 江苏华大天益电力科技有限公司 | Dynamic variable-frequency data collection method for monitoring wind-PV-ES hybrid power generation status |
| CN107070254A (en) * | 2017-04-13 | 2017-08-18 | 湖南科技大学 | A kind of Buck Boost matrix converters parameter optimization method and device |
| US20180032841A1 (en) * | 2016-08-01 | 2018-02-01 | Siemens Healthcare Gmbh | Medical Scanner Teaches Itself To Optimize Clinical Protocols And Image Acquisition |
| CN108173670A (en) * | 2016-12-07 | 2018-06-15 | 华为技术有限公司 | The method and apparatus for detecting network |
-
2020
- 2020-11-05 CN CN202011223995.XA patent/CN112307044A/en not_active Withdrawn
-
2021
- 2021-02-24 CN CN202110210085.6A patent/CN112732738B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104065168A (en) * | 2014-06-30 | 2014-09-24 | 江苏华大天益电力科技有限公司 | Dynamic variable-frequency data collection method for monitoring wind-PV-ES hybrid power generation status |
| US20180032841A1 (en) * | 2016-08-01 | 2018-02-01 | Siemens Healthcare Gmbh | Medical Scanner Teaches Itself To Optimize Clinical Protocols And Image Acquisition |
| CN108173670A (en) * | 2016-12-07 | 2018-06-15 | 华为技术有限公司 | The method and apparatus for detecting network |
| CN107070254A (en) * | 2017-04-13 | 2017-08-18 | 湖南科技大学 | A kind of Buck Boost matrix converters parameter optimization method and device |
Non-Patent Citations (1)
| Title |
|---|
| 洪意意 等: "基于多目标优化的自适应采集方法研究", 《CICC 2022》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112732738B (en) | 2023-04-18 |
| CN112307044A (en) | 2021-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112732738B (en) | Adaptive network data acquisition method based on multi-objective optimization and related equipment | |
| CN109002358B (en) | Mobile terminal software self-adaptive optimization scheduling method based on deep reinforcement learning | |
| JP2022003566A (en) | Correlation between thread strength and heep use amount for specifying stack trace in which heaps are stored up | |
| CN107547154B (en) | Method and device for establishing video traffic prediction model | |
| US20200184393A1 (en) | Method and apparatus for determining risk management decision-making critical values | |
| CN110852421A (en) | Model generation method and device | |
| WO2021129086A1 (en) | Traffic prediction method, device, and storage medium | |
| CN112149797B (en) | Neural network structure optimization method and device and electronic equipment | |
| CN111914516B (en) | Method, device, equipment and storage medium for generating network data prediction sequence | |
| CN112882889A (en) | Abnormality monitoring method, abnormality monitoring system, electronic device, and storage medium | |
| CN111371626A (en) | Bandwidth prediction method based on neural network | |
| EP2732387A1 (en) | Generation of recommendation values | |
| CN111813524A (en) | Task execution method and device, electronic equipment and storage medium | |
| CN109978172B (en) | Resource pool utilization rate prediction method and device based on extreme learning machine | |
| CN114157578B (en) | Network state prediction method and device | |
| CN113179224B (en) | Traffic scheduling method and device for content distribution network | |
| WO2023142351A1 (en) | Weight adjustment method and apparatus, and storage medium and electronic apparatus | |
| CN114500561B (en) | Power Internet of things network resource allocation decision-making method, system, equipment and medium | |
| CN116308450A (en) | Information generation and model training method, electronic equipment and storage medium | |
| US20220188383A1 (en) | Horizon-based smoothing of forecasting model | |
| CN113934542A (en) | Container resource adjusting method, system, medium and electronic terminal | |
| CN113505049A (en) | Cloud computing platform load prediction method based on firefly algorithm and BP neural network | |
| CN113342474A (en) | Method, device and storage medium for forecasting customer flow and training model | |
| CN110275809B (en) | Data fluctuation identification method and device and storage medium | |
| CN117076131B (en) | Task allocation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |