CN115086190B - Data processing method and device and computer storage medium - Google Patents
Data processing method and device and computer storage medium Download PDFInfo
- Publication number
- CN115086190B CN115086190B CN202210748229.8A CN202210748229A CN115086190B CN 115086190 B CN115086190 B CN 115086190B CN 202210748229 A CN202210748229 A CN 202210748229A CN 115086190 B CN115086190 B CN 115086190B
- Authority
- CN
- China
- Prior art keywords
- data
- acquisition device
- frequency
- acquisition
- historical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 33
- 238000004364 calculation method Methods 0.000 claims abstract description 58
- 239000002131 composite material Substances 0.000 claims abstract description 50
- 238000010801 machine learning Methods 0.000 claims description 52
- 230000000694 effects Effects 0.000 claims description 25
- 238000013480 data collection Methods 0.000 claims description 10
- 230000008685 targeting Effects 0.000 claims description 8
- 230000009467 reduction Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 11
- 238000000034 method Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 239000000498 cooling water Substances 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/022—Capturing of monitoring data by sampling
- H04L43/024—Capturing of monitoring data by sampling by adaptive sampling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0823—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability
- H04L41/0833—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability for reduction of network energy consumption
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The present disclosure relates to data processing methods and apparatuses, and computer-readable storage media, and relates to the field of data processing. The data processing method comprises the following steps: acquiring historical frequency of data acquisition by at least one acquisition device of a data center, wherein the historical frequency comprises a first historical frequency of data acquisition at a historical moment which is nearest to the current moment; according to the first historical frequency of the at least one acquisition device, aiming at reducing the composite average information Age (AOI) value of all data acquired by the at least one acquisition device, determining the current frequency of data acquisition by each acquisition device; and controlling each acquisition device to acquire data according to the current frequency corresponding to the acquisition device, wherein the data acquired by each acquisition device are used for data calculation, and the data calculation result corresponding to at least one acquisition device is used for determining the energy-saving operation for the data center. According to the present disclosure, timeliness of energy saving processing for a data center can be improved.
Description
Technical Field
The present disclosure relates to the field of data processing, and in particular, to a data processing method and apparatus, and a computer readable medium.
Background
In the process of using the IDC (Internet Data Center ) machine room energy-saving scheme, the sensor needs to acquire a series of time-varying environmental data such as the current cooling water temperature, the machine room temperature, the outdoor temperature and the like in real time. The magnitude of the time difference that the relevant time-varying data is collected and updated to the control system is the key to decide whether the control system can make accurate energy-saving decisions. If the data is congested due to too slow update frequency or too fast update frequency, the real-time performance of the control decision is greatly affected, and effective energy saving cannot be achieved. Therefore, how to improve the timeliness of data transmission between the sensor and the controller is a key to maximize the energy saving strategy effect.
In the related art, the frequency of data acquisition by the sensor is adjusted empirically.
Disclosure of Invention
In the related art, the frequency is adjusted according to experience, objectivity is lacked, and timeliness of data acquired by the acquisition equipment cannot be effectively improved, so that timeliness of energy-saving processing of a data center cannot be effectively improved.
In view of the above technical problems, the present disclosure proposes a solution that can improve timeliness of energy-saving processing for a data center.
According to a first aspect of the present disclosure, there is provided a data processing method comprising: acquiring historical frequency of data acquisition by at least one acquisition device of a data center, wherein the historical frequency comprises a first historical frequency of data acquisition at a historical moment which is nearest to the current moment; according to the first historical frequency of the at least one acquisition device, aiming at reducing the composite average information Age (AOI) value of all data acquired by the at least one acquisition device, determining the current frequency of data acquisition by each acquisition device; and controlling each acquisition device to acquire data according to the current frequency corresponding to the acquisition device, wherein the data acquired by each acquisition device are used for data calculation, and the data calculation result corresponding to at least one acquisition device is used for determining the energy-saving operation for the data center.
In some embodiments, based on the first historical frequency of the at least one acquisition device, targeting reducing the composite average information age, AOI, value for all data acquired by the at least one acquisition device, determining the current frequency at which each acquisition device performs data acquisition includes: and determining the current frequency of data acquisition by each acquisition device by using a machine learning model according to the first historical frequency of the at least one acquisition device, wherein the machine learning model aims at reducing the composite average information Age (AOI) value of all data acquired by the at least one acquisition device.
In some embodiments, the data computation is performed by a computing device, each acquisition device transmitting data it acquires to a data computation queue of the computing device, determining a current frequency of data acquisition by each acquisition device using a machine learning model comprising: for data acquired by each acquisition device at a historical moment which is closest to the current moment, acquiring at least one of a historical queue growth speed of a data calculation queue in which the data is positioned in a preset duration and a historical energy-saving effect of a data calculation result corresponding to the data for energy-saving operation; and determining the current frequency of data acquisition by each acquisition device by using the machine learning model according to at least one of the historical energy saving effect and the historical queue growth speed corresponding to the historical moment closest to the current moment and the first historical frequency of the at least one acquisition device, wherein when the historical queue growth speed is greater than a growth speed threshold, the input of the machine learning model comprises the historical queue growth speed.
In some embodiments, the data processing method further comprises: acquiring time information from the data acquisition start to the data calculation end of the data acquired by each acquisition device at the historical frequency and the current frequency respectively, wherein the historical frequency also comprises a second historical acquisition frequency for carrying out data acquisition at other historical moments except the historical moment closest to the current moment; for each of the historical frequency and the current frequency, determining an instantaneous AOI value of data acquired by each acquisition device at each frequency according to time information corresponding to each frequency; determining an average value of the respective instantaneous AOI values corresponding to the respective frequencies of the respective acquisition devices; determining the composite average AOI value according to the average value of each instantaneous AOI value corresponding to the at least one acquisition device; and training the machine learning model by taking the composite average information age AOI value as a target.
In some embodiments, determining the current frequency at which each acquisition device performs data acquisition comprises: acquiring a historical AOI weight value of each acquisition device in the at least one acquisition device at a historical moment closest to the current moment; determining the current frequency and the current AOI weight value of each acquisition device for data acquisition by using a machine learning model according to the first historical frequency of the at least one acquisition device and the historical AOI weight value at the historical moment closest to the current moment; determining the composite average AOI value includes: and carrying out weighting operation on the average AOI value corresponding to the at least one acquisition device according to the current AOI weight value corresponding to each acquisition device to obtain the composite average AOI value.
In some embodiments, the data computation is performed by a computing device, each acquisition device transmitting its acquired data to a data computation queue of the computing device, targeting a reduction in composite average AOI values for all data acquired by the at least one acquisition device, training the machine learning model comprising: acquiring the current queue growth speed of the data calculation queue in a preset duration; and training the machine learning model with the aim of reducing the composite average AOI value and reducing the queue growth rate of the data calculation queue under the condition that the current queue growth rate is greater than a growth rate threshold.
In some embodiments, the data collected by each collection device at each frequency has a collection time stamp, and the data processing method further comprises: and deleting the data with the acquisition time stamp earlier than the time stamp threshold value in the data calculation queue under the condition that the current queue growth speed is greater than the growth speed threshold value.
In some embodiments, the data acquired by each acquisition device at each frequency has an acquisition time stamp, and determining an average of the respective instantaneous AOI values corresponding to the respective frequencies of each acquisition device comprises: for each acquisition device, integrating the instantaneous AOI value corresponding to each frequency of each acquisition device in time according to the acquisition time stamp of the data acquired by each acquisition device at each frequency to obtain a total AOI value corresponding to each acquisition device; and determining the average value of the instantaneous AOI values corresponding to the frequencies of each acquisition device according to the ratio of the total AOI value to the total number of the acquisition time stamps.
In some embodiments, the data calculation is performed by a computing device, each acquisition device transmitting data acquired by it to the computing device, the time information of the data acquired by each acquisition device at each frequency from the beginning of the data acquisition to the end of the data calculation comprising: at least one of a first time, a second time, a third time, a fourth time, and a fifth time, wherein the first time characterizes a time when data is acquired at the each frequency, the second time characterizes a time when data acquired at the each frequency is transmitted from an acquisition device to the computing device, the third time characterizes a time when data acquired at the each frequency waits for data calculation at the computing device, the fourth time characterizes a time when data is acquired at the computing device at the each frequency, and the fifth time characterizes a time interval between a time when data acquired at the each frequency completes data calculation at the computing device and a time when the computing device has last completed data calculation.
In some embodiments, determining the instantaneous AOI value for the data acquired by each acquisition device at said each frequency comprises: and determining the instantaneous AOI value of the data acquired by each acquisition device at each frequency according to the sum of the first time, the second time, the third time, the fourth time and the fifth time corresponding to the data acquired by each acquisition device at each frequency.
In some embodiments, targeting the reduction of the composite average information age AOI value, training the machine learning model includes: acquiring a current energy-saving effect of energy-saving operation at the current moment corresponding to the at least one acquisition device; and training the machine learning model with the aim of reducing the composite average AOI value and improving the energy-saving effect.
In some embodiments, the data computation is performed by a computing device, each acquisition device transmitting its acquired data to a data computation queue of the computing device, with the goal of reducing the composite average AOI value and improving the energy savings effect, training the machine learning model comprising: acquiring the current queue growth speed of a data calculation queue in which the data acquired at the current moment are located in a preset time length; and under the condition that the current queue growth speed is larger than a growth speed threshold, training the machine learning model with the aim of reducing the composite average AOI value, improving the energy saving effect and reducing the queue growth speed of the data calculation queue.
According to a second aspect of the present disclosure, there is provided a data processing apparatus comprising: the acquisition module is configured to acquire historical frequency of data acquisition by at least one acquisition device of the data center, wherein the historical frequency comprises a first historical frequency of data acquisition at a historical moment closest to the current moment; a determining module configured to determine, according to a first historical frequency of the at least one collecting device, a current frequency of data collection by each collecting device with the aim of reducing a composite average information age AOI value of all data collected by the at least one collecting device; the control module is configured to control each acquisition device to acquire data according to the current frequency corresponding to the acquisition device, wherein the data acquired by each acquisition device are used for data calculation, and the data calculation result corresponding to the at least one acquisition device is used for determining energy-saving operation for the data center.
According to a third aspect of the present disclosure, there is provided a data processing apparatus comprising: a memory; and a processor coupled to the memory, the processor configured to perform the data processing method of any of the embodiments described above based on instructions stored in the memory.
According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement a data processing method according to any of the embodiments described above.
In the above embodiment, the timeliness of the energy saving process for the data center can be improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description, serve to explain the principles of the disclosure.
The disclosure may be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings in which:
FIG. 1 is a flow chart illustrating a data processing method according to some embodiments of the present disclosure;
FIG. 2 is a block diagram illustrating a data processing apparatus according to some embodiments of the present disclosure;
FIG. 3 is a block diagram illustrating a data processing apparatus according to further embodiments of the present disclosure;
FIG. 4 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.
Detailed Description
Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless it is specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective parts shown in the drawings are not drawn in actual scale for convenience of description.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any specific values should be construed as merely illustrative, and not a limitation. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.
Fig. 1 is a flow chart illustrating a data processing method according to some embodiments of the present disclosure.
As shown in fig. 1, the data processing method includes steps S110 to S130.
In step S110, a historical frequency of data collection by at least one collection device of the data center is obtained. The historical frequency includes a first historical frequency for data collection at a historical time that is closest to the current time. The history time closest to the current time is, for example, a time preceding the current time. In some embodiments, the acquisition device includes a sensor or the like. In some embodiments, there are multiple historical frequencies. The sensor can acquire a series of time-varying environmental data such as cooling water temperature, machine room temperature, outdoor temperature and the like of the data center for determining energy-saving operation.
In step S120, the current frequency of data acquisition by each acquisition device is determined based on the first historical frequency of the at least one acquisition device, targeting a reduction in the composite average AOI (Age of Information, information age) value of all data acquired by the at least one acquisition device. In some embodiments, the composite average AOI value reflects freshness or timeliness or real-time of all data acquired by all acquisition devices.
In some embodiments, a current frequency of data acquisition by each acquisition device may be determined using a machine learning model based on a first historical frequency of at least one acquisition device. The machine learning model targets reducing a composite average AOI value for all data acquired by the at least one acquisition device. In training the machine learning model, a loss value of the loss function is determined based on the composite average AOI value. For example, the goal is to reduce the composite average AOI value below the composite average AOI threshold or to minimize the composite average AOI value.
In some embodiments, the data computation is performed by the computing devices, with each acquisition device transmitting the data it acquires to a data computation queue of the computing device. Determining the current frequency of data acquisition by each acquisition device using a machine learning model includes the following steps.
Firstly, for data acquired by each acquisition device at a historical moment which is closest to the current moment, acquiring at least one of a historical queue growth speed of a data calculation queue in which the data is positioned in a preset duration and a historical energy-saving effect of a data calculation result corresponding to the data for energy-saving operation.
And then, determining the current frequency of data acquisition by each acquisition device by using a machine learning model according to at least one of the historical energy saving effect and the historical queue growth speed corresponding to the historical moment closest to the current moment and the first historical frequency of at least one acquisition device. In the event that the historical queue growth rate is greater than the growth rate threshold, the input to the machine learning model includes the historical queue growth rate. The accuracy of energy-saving processing is further improved by increasing at least one of queue growth speed feedback and historical energy-saving effect feedback at historical moments.
In step S130, each acquisition device is controlled to perform data acquisition according to the current frequency corresponding to the acquisition device. The data collected by each collection device is used for data calculation, and the data calculation results corresponding to at least one collection device are used for determining energy-saving operation for the data center. In some embodiments, the data calculation corresponding to the at least one acquisition device is input into an energy-saving algorithm model resulting in an energy-saving strategy comprising one or more energy-saving operations.
In the above embodiment, for each acquisition device, before each data acquisition, the composite average information age AOI value of all the data acquired by at least one acquisition device is reduced, the acquisition frequency of each acquisition device is updated, and the timeliness of the data acquired by the acquisition device is improved, so that the data for determining the energy-saving operation is always valid in real time, thereby effectively improving the timeliness of the energy-saving processing for the data center and improving the accuracy of the energy-saving processing.
In some embodiments, the data processing method further comprises the following steps.
Firstly, time information of data acquired by each acquisition device from the beginning of data acquisition to the end of data calculation at a historical frequency and a current frequency respectively is acquired. The history frequency further includes a second history acquisition frequency at which data is acquired at a history time other than the history time closest to the current time.
Next, for each of the historical frequency and the current frequency, an instantaneous AOI value of data acquired at each frequency by each acquisition device is determined from time information corresponding to each frequency.
In some embodiments, taking as an example that the data calculation is performed by the computing device and each collecting device transmits the data collected by it to the computing device, the time information of the data collected by each collecting device at each frequency from the beginning of the data collection to the end of the data calculation includes: at least one of the first time, the second time, the third time, the fourth time, and the fifth time. The first time characterizes the time at which data acquisition occurs at each frequency. The second time characterizes a time at which data acquired at each frequency is transmitted from the acquisition device to the computing device. The third time characterizes the time at which the data acquired at each frequency waits for data computation at the computing device. The fourth time characterizes the time at which the data acquired at each frequency was calculated at the computing device. The fifth time characterizes a time interval between a time at which the computing device completed the data calculation and a time at which the computing device completed the data calculation last time for the data acquired at each frequency.
In some embodiments, the instantaneous AOI value of the data acquired by each acquisition device at each frequency is determined from a sum of a first time, a second time, a third time, a fourth time, and a fifth time corresponding to the data acquired by each acquisition device at each frequency.
For example, the instantaneous AOI value for data with an acquisition timestamp of i may be expressed as AOI i ＝T i +W i +C i +x i 。T i Is the sum of the first time and the second time, W i 、C i And x i The third time, the fourth time, and the fifth time are represented, respectively.
Again, an average of the respective instantaneous AOI values corresponding to the respective frequencies of each acquisition device is determined.
In some embodiments, taking an example that each acquisition device takes data acquired at each frequency as an acquisition time stamp, for each acquisition device, integrating instantaneous AOI values corresponding to each frequency of each acquisition device in time according to the acquisition time stamp of the data acquired at each frequency by each acquisition device to obtain a total AOI value corresponding to each acquisition device; an average of the respective instantaneous AOI values corresponding to the respective frequencies of each acquisition device is determined from the ratio of the total AOI value to the total number of acquisition timestamps.
For example, the average value of the respective instantaneous AOI values corresponding to the respective frequencies of a certain acquisition device is expressed asWhere t is the total number of acquisition time stamps. In some embodiments, the acquisition time stamp may be marked 0, 1, …, t.
Then, a composite average AOI value is determined from an average of the respective instantaneous AOI values corresponding to the at least one acquisition device.
Finally, the machine learning model is trained with the goal of reducing the composite average information age AOI value. Training the machine learning model, that is, updating weight parameters and bias parameters of the machine learning model, and the like.
In some embodiments, determining the current frequency at which each acquisition device performs data acquisition may be accomplished as follows.
First, a historical AOI weight value for each of at least one acquisition device at a historical time closest to a current time is obtained. The historical AOI weight value may also be used as input to a machine learning model along with the historical energy saving effect and historical queue growth rate in the previous embodiments. In some embodiments, a last historical AOI weight value for each of the at least one acquisition device is obtained. The historical AOI weight value at the historical moment closest to the current moment is the instantaneous AOI weight value at the historical moment closest to the current moment.
The historical AOI weight value for each acquisition device characterizes the degree of importance of the timeliness of the data acquired by the acquisition device at a historical time for determining energy-saving operation. The higher the historical AOI weight value, the higher the timeliness requirement of the data acquired by the corresponding acquisition equipment, and the higher the frequency of data acquisition.
The AOI weight value of each acquisition device is used to control the data acquisition frequency of that acquisition device. For example, the data collected is uploaded or transmitted to a computing device or data processor, and the data collection frequency is consistent with the data transmission or uploading frequency. If the weight value is higher, the current change of the collected time-varying data is faster or the time is more sensitive, the corresponding collection equipment can accelerate the data collection speed and the data uploading priority, the AOI value of the data is ensured to be lower, and the energy-saving algorithm for determining the energy-saving operation can obtain the data collection result updated in the most real time. If the weight value is lower, the data change is slow at present or the energy-saving algorithm does not pay attention to the data result temporarily, and the corresponding acquisition equipment reduces the data acquisition and data uploading speed.
Then, a current frequency and a current AOI weight value of each acquisition device for data acquisition are determined by using a machine learning model according to the first historical frequency of at least one acquisition device and the historical AOI weight value at a historical moment nearest to the current moment.
In this case, the average AOI value corresponding to at least one acquisition device may be weighted according to the current AOI weight value corresponding to each acquisition device, to obtain a composite average information age AOI value. The composite average AOI value is constantly changing as the acquisition device performs data acquisition operations.
In the above embodiment, the machine learning model may predict not only the current frequency of data acquisition but also the current AOI weight value, so as to adaptively update the target of machine learning, thereby further improving the prediction accuracy of the data acquisition frequency, and further improving the timeliness of the data acquired by the acquisition device, so that the data for determining the energy-saving operation is always valid in real time on the whole, and thus effectively improving the timeliness of the energy-saving processing for the data center.
In some embodiments, the data computation is performed by the computing devices, with each acquisition device transmitting the data it acquires to a data computation queue of the computing device. The computing device obtains data from the data computation queue in a first-in first-out manner for data computation. For example, a machine learning model may be trained by targeting a reduction in the composite average AOI value of all data acquired by at least one acquisition device in the following manner.
And firstly, acquiring the current queue growth speed of the data calculation queue in a preset time length.
Then, in the event that the current queue growth rate is greater than the growth rate threshold, a machine learning model is trained with the goal of reducing the composite average AOI value and reducing the queue growth rate of the data computation queue.
In the case where the number of acquisition devices is greater than the number of computing devices, one computing device will process data acquired by multiple acquisition devices simultaneously. According to queuing theory, it can be known that the computing device is likely to have queuing conditions due to fluctuation of transmission intervals of data acquired by the acquisition device. In the above embodiment, by training the machine learning model with the goal of reducing the composite average AOI value and reducing the queue growth speed at the same time, the freshness of the collected data can be improved, the data redundancy in the data calculation queue can be reduced, the processing efficiency of the computing device can be improved, and the situation that the computing device processes the outdated data can be reduced, so that the invalid operation of the energy-saving processing can be reduced, and the effectiveness and the instantaneity of the energy-saving processing of the data center can be aimed at.
In some embodiments, the data acquired by each acquisition device at each frequency has an acquisition timestamp. In the event that the current queue growth rate is greater than the growth rate threshold, deleting data in the data calculation queue whose acquisition timestamp is earlier than the timestamp threshold. By the method, the data redundancy in the data calculation queue can be reduced, the processing efficiency of the computing equipment is improved, the situation that the computing equipment processes outdated data is reduced, and therefore invalid operation of energy-saving processing is reduced, and the effectiveness and instantaneity of energy-saving processing of a data center can be aimed.
In some embodiments, training the machine learning model with the goal of reducing the composite average information age AOI value may also be accomplished as follows.
First, an energy saving effect of an energy saving operation corresponding to at least one acquisition device is acquired. In some embodiments, the energy saving effect may be measured using an energy saving index.
Then, the machine learning model is trained with the aim of reducing the composite average AOI value and improving the energy saving effect. In this way, the timeliness of the acquired data is improved, and meanwhile, the energy-saving effect is improved.
In some embodiments, taking as an example a data calculation performed by a computing device and each acquisition device transmitting its acquired data to a data calculation queue of the computing device, training a machine learning model with the goal of reducing composite average AOI values and improving energy savings includes the following steps.
Firstly, the current queue growth speed of the data calculation queue in a preset duration is obtained.
Then, under the condition that the current queue growth speed is larger than the growth speed threshold, the machine learning model is trained with the aim of reducing the composite average AOI value, improving the energy saving effect and reducing the queue growth speed of the data calculation queue. The machine learning model is trained through multi-aspect feedback, and the accuracy of the machine learning model is improved, so that the timeliness of collected data, the energy-saving effect and the congestion degree of data calculation are balanced, and the energy-saving effect is improved from multiple aspects. The target for training the machine learning model can be converted into calculation of the loss function, and then the loss value of the loss function is updated by utilizing algorithms such as gradient descent and the like, so that the purpose of training the machine learning model is realized.
In some embodiments, the machine learning model may be a reinforcement learning model. The Action is selected to determine the current frequency and the current AOI weight value, the State is selected to be a related environment variable, and the Reward can select a composite average AOI value and the ratio of the energy-saving actual effect to the theoretical value. The machine learning model may also be other models.
Fig. 2 is a block diagram illustrating a data processing apparatus according to some embodiments of the present disclosure.
As shown in fig. 2, the data processing apparatus 2 includes an acquisition module 21, a determination module 22, and a control module 23.
The acquisition module 21 is configured to acquire a historical frequency of data acquisition by at least one acquisition device of the data center, for example, performing step S110 as shown in fig. 1. The historical frequency includes a first historical frequency for data collection at a historical time that is closest to the current time.
The determining module 22 is configured to determine the current frequency of data acquisition by each acquisition device, for example performing step S120 as shown in fig. 1, with the aim of reducing the composite average information age AOI value of all data acquired by the at least one acquisition device, based on the first historical frequency of the at least one acquisition device.
The control module 23 is configured to control each acquisition device to perform data acquisition at its corresponding current frequency, for example, to perform step S130 shown in fig. 1. The data collected by each collection device is used for data calculation, and the data calculation results corresponding to at least one collection device are used for determining energy-saving operation for the data center.
Fig. 3 is a block diagram illustrating a data processing apparatus according to further embodiments of the present disclosure.
As shown in fig. 3, the data processing apparatus 3 includes a memory 31; and a processor 32 coupled to the memory 31. The memory 31 is used for storing instructions for executing corresponding embodiments of the data processing method. The processor 32 is configured to perform the data processing method in any of the embodiments of the present disclosure based on instructions stored in the memory 31.
FIG. 4 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.
As shown in FIG. 4, computer system 40 may be in the form of a general purpose computing device. Computer system 40 includes a memory 410, a processor 420, and a bus 400 that connects the various system components.
Memory 410 may include, for example, system memory, non-volatile storage media, and the like. The system memory stores, for example, an operating system, application programs, boot Loader (Boot Loader), and other programs. The system memory may include volatile storage media, such as Random Access Memory (RAM) and/or cache memory. The non-volatile storage medium stores, for example, instructions of a corresponding embodiment that perform at least one of the data processing. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, and the like.
Processor 420 may be implemented as discrete hardware components such as a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gates or transistors, and the like. Accordingly, each of the modules, such as the judgment module and the determination module, may be implemented by a Central Processing Unit (CPU) executing instructions of the corresponding steps in the memory, or may be implemented by a dedicated circuit that performs the corresponding steps.
Bus 400 may employ any of a variety of bus architectures. For example, bus structures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.
Computer system 40 may also include an input-output interface 430, a network interface 440, a storage interface 450, and the like. These interfaces 430, 440, 450 and the memory 410 and processor 420 may be connected by a bus 400. The input output interface 430 may provide a connection interface for input output devices such as a display, mouse, keyboard, etc. Network interface 440 provides a connection interface for various networking devices. The storage interface 450 provides a connection interface for external storage devices such as floppy disks, U disks, SD cards, and the like.
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable apparatus to produce a machine, such that the instructions, which execute via the processor, create means for implementing the functions specified in the flowchart and/or block diagram block or blocks.
These computer readable program instructions may also be stored in a computer readable memory that can direct a computer to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and/or block diagram block or blocks.
The present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.
By the data processing method, the data processing device and the computer storage medium in the embodiment, the timeliness of energy-saving processing for the data center can be improved.
So far, the data processing method and apparatus, computer-readable storage medium according to the present disclosure have been described in detail. In order to avoid obscuring the concepts of the present disclosure, some details known in the art are not described. How to implement the solutions disclosed herein will be fully apparent to those skilled in the art from the above description.
Claims (15)
1. A data processing method, comprising:
acquiring historical frequency of data acquisition by at least one acquisition device of a data center, wherein the historical frequency comprises a first historical frequency of data acquisition at a historical moment which is nearest to the current moment;
according to the first historical frequency of the at least one acquisition device, aiming at reducing the composite average information Age (AOI) value of all data acquired by the at least one acquisition device, determining the current frequency of data acquisition by each acquisition device;
and controlling each acquisition device to acquire data according to the current frequency corresponding to the acquisition device, wherein the data acquired by each acquisition device are used for data calculation, and the data calculation result corresponding to at least one acquisition device is used for determining the energy-saving operation for the data center.
2. The data processing method of claim 1, wherein determining the current frequency of data acquisition by each acquisition device, based on the first historical frequency of the at least one acquisition device, targeting a reduction in composite average information age, AOI, value for all data acquired by the at least one acquisition device, comprises:
and determining the current frequency of data acquisition by each acquisition device by using a machine learning model according to the first historical frequency of the at least one acquisition device, wherein the machine learning model aims at reducing the composite average information Age (AOI) value of all data acquired by the at least one acquisition device.
3. The data processing method of claim 2, wherein the data computation is performed by a computing device, each acquisition device transmitting data it acquires to a data computation queue of the computing device, determining a current frequency of data acquisition by each acquisition device using a machine learning model comprising:
for data acquired by each acquisition device at a historical moment which is closest to the current moment, acquiring at least one of a historical queue growth speed of a data calculation queue in which the data is positioned in a preset duration and a historical energy-saving effect of a data calculation result corresponding to the data for energy-saving operation;
and determining the current frequency of data acquisition by each acquisition device by using the machine learning model according to at least one of the historical energy saving effect and the historical queue growth speed corresponding to the historical moment closest to the current moment and the first historical frequency of the at least one acquisition device, wherein when the historical queue growth speed is greater than a growth speed threshold, the input of the machine learning model comprises the historical queue growth speed.
4. The data processing method of claim 2, further comprising:
acquiring time information from the data acquisition start to the data calculation end of the data acquired by each acquisition device at the historical frequency and the current frequency respectively, wherein the historical frequency also comprises a second historical acquisition frequency for carrying out data acquisition at other historical moments except the historical moment closest to the current moment;
for each of the historical frequency and the current frequency, determining an instantaneous AOI value of data acquired by each acquisition device at each frequency according to time information corresponding to each frequency;
determining an average value of the respective instantaneous AOI values corresponding to the respective frequencies of the respective acquisition devices;
determining the composite average AOI value according to the average value of each instantaneous AOI value corresponding to the at least one acquisition device;
and training the machine learning model by taking the composite average information age AOI value as a target.
5. The data processing method according to claim 4, wherein,
determining the current frequency of data acquisition by each acquisition device includes:
acquiring a historical AOI weight value of each acquisition device in the at least one acquisition device at a historical moment closest to the current moment;
determining the current frequency and the current AOI weight value of each acquisition device for data acquisition by using a machine learning model according to the first historical frequency of the at least one acquisition device and the historical AOI weight value at the historical moment closest to the current moment; determining the composite average AOI value includes:
and carrying out weighting operation on the average AOI value corresponding to the at least one acquisition device according to the current AOI weight value corresponding to each acquisition device to obtain the composite average AOI value.
6. The data processing method of claim 4 or 5, wherein the data computation is performed by a computing device, each acquisition device transmitting data acquired by it to a data computation queue of the computing device, targeting a reduction in composite average AOI values for all data acquired by the at least one acquisition device, training the machine learning model comprising:
acquiring the current queue growth speed of the data calculation queue in a preset duration;
and training the machine learning model with the aim of reducing the composite average AOI value and reducing the queue growth rate of the data calculation queue under the condition that the current queue growth rate is greater than a growth rate threshold.
7. The data processing method of claim 6, wherein the data acquired by each acquisition device at each frequency has an acquisition time stamp, the data processing method further comprising:
and deleting the data with the acquisition time stamp earlier than the time stamp threshold value in the data calculation queue under the condition that the current queue growth speed is greater than the growth speed threshold value.
8. The data processing method of claim 4, wherein the data acquired by each acquisition device at each frequency has an acquisition time stamp, and determining an average of the respective instantaneous AOI values corresponding to the respective frequencies of each acquisition device comprises:
for each acquisition device, integrating the instantaneous AOI value corresponding to each frequency of each acquisition device in time according to the acquisition time stamp of the data acquired by each acquisition device at each frequency to obtain a total AOI value corresponding to each acquisition device;
and determining the average value of the instantaneous AOI values corresponding to the frequencies of each acquisition device according to the ratio of the total AOI value to the total number of the acquisition time stamps.
9. The data processing method according to claim 4, wherein the data calculation is performed by a computing device, each of the collecting devices transmitting data collected by it to the computing device, time information of data collected by each of the collecting devices at each frequency from a start of data collection to an end of data calculation including: at least one of a first time, a second time, a third time, a fourth time, and a fifth time, wherein the first time characterizes a time when data is acquired at the each frequency, the second time characterizes a time when data acquired at the each frequency is transmitted from an acquisition device to the computing device, the third time characterizes a time when data acquired at the each frequency waits for data calculation at the computing device, the fourth time characterizes a time when data is acquired at the computing device at the each frequency, and the fifth time characterizes a time interval between a time when data acquired at the each frequency completes data calculation at the computing device and a time when the computing device has last completed data calculation.
10. The data processing method of claim 9, wherein determining the instantaneous AOI value of the data acquired by each acquisition device at the each frequency comprises:
and determining the instantaneous AOI value of the data acquired by each acquisition device at each frequency according to the sum of the first time, the second time, the third time, the fourth time and the fifth time corresponding to the data acquired by each acquisition device at each frequency.
11. The data processing method of claim 4, wherein training the machine learning model with the goal of reducing the composite average information age AOI value comprises:
acquiring a current energy-saving effect of energy-saving operation at the current moment corresponding to the at least one acquisition device;
and training the machine learning model with the aim of reducing the composite average AOI value and improving the energy-saving effect.
12. The data processing method of claim 11, wherein the data computation is performed by a computing device, each acquisition device transmitting its acquired data to a data computation queue of the computing device, targeting a reduction in the composite average AOI value and an increase in the energy savings effect, training the machine learning model comprising:
acquiring the current queue growth speed of a data calculation queue in which the data acquired at the current moment are located in a preset time length;
and under the condition that the current queue growth speed is larger than a growth speed threshold, training the machine learning model with the aim of reducing the composite average AOI value, improving the energy saving effect and reducing the queue growth speed of the data calculation queue.
13. A data processing apparatus comprising:
the acquisition module is configured to acquire historical frequency of data acquisition by at least one acquisition device of the data center, wherein the historical frequency comprises a first historical frequency of data acquisition at a historical moment closest to the current moment;
a determining module configured to determine, according to a first historical frequency of the at least one collecting device, a current frequency of data collection by each collecting device with the aim of reducing a composite average information age AOI value of all data collected by the at least one collecting device;
the control module is configured to control each acquisition device to acquire data according to the current frequency corresponding to the acquisition device, wherein the data acquired by each acquisition device are used for data calculation, and the data calculation result corresponding to the at least one acquisition device is used for determining energy-saving operation for the data center.
14. A data processing apparatus comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the data processing method of any of claims 1 to 12 based on instructions stored in the memory.
15. A computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement a data processing method as claimed in any one of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210748229.8A CN115086190B (en) | 2022-06-29 | 2022-06-29 | Data processing method and device and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210748229.8A CN115086190B (en) | 2022-06-29 | 2022-06-29 | Data processing method and device and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115086190A CN115086190A (en) | 2022-09-20 |
| CN115086190B true CN115086190B (en) | 2024-03-01 |
Family
ID=83256347
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210748229.8A Active CN115086190B (en) | 2022-06-29 | 2022-06-29 | Data processing method and device and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115086190B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114039918A (en) * | 2021-10-09 | 2022-02-11 | 广东技术师范大学 | Information age optimization method and device, computer equipment and storage medium |
| CN114374949A (en) * | 2021-12-31 | 2022-04-19 | 东莞理工学院 | Power control mechanism based on information freshness optimization in Internet of vehicles |
| CN114675845A (en) * | 2022-03-29 | 2022-06-28 | 广东技术师范大学 | Information age optimization method and device, computer equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10691082B2 (en) * | 2017-12-05 | 2020-06-23 | Cisco Technology, Inc. | Dynamically adjusting sample rates based on performance of a machine-learning based model for performing a network assurance function in a network assurance system |
-
2022
- 2022-06-29 CN CN202210748229.8A patent/CN115086190B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114039918A (en) * | 2021-10-09 | 2022-02-11 | 广东技术师范大学 | Information age optimization method and device, computer equipment and storage medium |
| CN114374949A (en) * | 2021-12-31 | 2022-04-19 | 东莞理工学院 | Power control mechanism based on information freshness optimization in Internet of vehicles |
| CN114675845A (en) * | 2022-03-29 | 2022-06-28 | 广东技术师范大学 | Information age optimization method and device, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| Sihua Wang等.Distributed Reinforcement Learning for Age of Information Minimization in Real-Time IoT Systems.《IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING》.2022,第第16卷卷(第第3期期),501-515. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115086190A (en) | 2022-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2023093820A1 (en) | Device control optimization method, display platform, cloud server, and storage medium | |
| CN113989561B (en) | Parameter aggregation updating method, device and system based on asynchronous federal learning | |
| CN103853918B (en) | A kind of cloud computing server dispatching method based on free time prediction | |
| CN111615589A (en) | Method and device for the coordinated control of wind turbines of a wind park | |
| JP2020154785A (en) | Prediction method, prediction program, and model learning method | |
| US20140258382A1 (en) | Application congestion control | |
| CN111582604A (en) | Data processing method and device, electronic device, and medium | |
| CN113778691B (en) | Task migration decision method, device and system | |
| CN113687960B (en) | Edge computing intelligent caching method based on deep reinforcement learning | |
| CN113222118B (en) | Neural network training method, apparatus, electronic device, medium, and program product | |
| CN115915708B (en) | Refrigeration equipment control parameter prediction method and device, electronic equipment and storage medium | |
| CN112488183A (en) | Model optimization method and device, computer equipment and storage medium | |
| CN111030850B (en) | SCADA system data acquisition period control method and device | |
| CN113887748B (en) | Online federal learning task allocation method and device, and federal learning method and system | |
| CN115086190B (en) | Data processing method and device and computer storage medium | |
| CN113791538B (en) | Control method, control device and control system of machine room equipment | |
| CN113821903A (en) | Temperature control method and device, modular data center and storage medium | |
| CN111505944A (en) | Energy-saving control strategy learning method, and method and device for realizing air conditioning energy control | |
| CN109379747A (en) | The deployment of wireless network multi-controller and resource allocation methods and device | |
| CN116795066A (en) | Communication data processing method, system, server and medium of remote IO module | |
| JP7287835B2 (en) | Model predictive control system, information processing device, program, and model predictive control method | |
| CN117332896A (en) | New energy small time scale power prediction method and system for multilayer integrated learning | |
| CN112486683B (en) | Processor control method, control apparatus, and computer-readable storage medium | |
| CN114866489A (en) | Congestion control method and device and training method and device of congestion control model | |
| CN107710168B (en) | Information processing unit and information processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |