WO2021147319A1

WO2021147319A1 - Data processing method, apparatus, device, and medium

Info

Publication number: WO2021147319A1
Application number: PCT/CN2020/111993
Authority: WO
Inventors: 霍罗威茨·夏伊; 埃瑞恩·雅尔; 佩雷斯·诺阿姆; 王琛
Original assignee: 华为技术有限公司
Priority date: 2020-01-23
Filing date: 2020-08-28
Publication date: 2021-07-29
Also published as: CN113162960A

Abstract

The present application provides a data processing method. Said method comprises: a first device acquiring a plurality of pieces of data, then the first device obtaining a target fitting model on the basis of the plurality of pieces of data, and then the first device sending the target fitting model to a second device, the target fitting model being used to recover at least one of the plurality of pieces of data. In this way, the amount of data to be transmitted and the amount of data to be stored by a second device can be reduced, thereby relieving the data transmission pressure and the data storage pressure.

Description

Data processing method, device, equipment and medium

This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office of China, the application number is 202010076999.3, and the invention title is "a data processing method, device, equipment and medium" on January 23, 2020. The entire content of the application is approved The reference is incorporated in this application.

Technical field

This application relates to the field of computer technology, and in particular to a data processing method, device, device, and computer-readable storage medium.

Background technique

With the advent of the information age, massive amounts of data have also been produced. In order to make full use of the above-mentioned massive data, it is often necessary to transmit and store the data. Taking the Internet of Things (IoT) as an example, more and more devices such as TVs, air conditioners, speakers, routers, cameras, etc. can be independently addressed to form an interconnected network, thereby realizing intelligent identification of devices , Positioning, tracking, monitoring and management.

In order to realize intelligent perception, identification, positioning, tracking, monitoring or management, IoT devices generate a large amount of time series data. The so-called time series data refers to the index data that is calculated according to time. The time series data may be resource utilization rate, electrocardiogram, stock price, and so on. In practical applications, the amount of time-series data generated by IoT devices every minute can reach hundreds of millions, which leads to greater pressure on data transmission and storage.

Based on this, the industry urgently needs to provide a data processing method to relieve the pressure of data transmission and storage.

Summary of the invention

This application provides a data processing method, which realizes data compression by performing model fitting on data, and solves the problem of high data transmission and storage pressure. This application also provides corresponding devices, equipment, computer-readable storage media, and computer program products.

In the first aspect, this application provides a data processing method. Specifically, the first device as the data sender can obtain multiple data, and then obtain the target fitting model based on the multiple data, and then send the aforementioned target fitting model to the second device, instead of directly sending multiple data, the second The device can recover at least one of the multiple data through the above-mentioned target fitting model according to requirements. In this way, the amount of transmitted data and the amount of data stored by the second device can be reduced, thereby alleviating the pressure of data transmission and data storage.

In some possible implementation manners, the first device may obtain the target fitting model in the following manner. Specifically, the first device may perform model fitting on multiple fitting models according to multiple data, so as to select one fitting model from the multiple fitting models as the target fitting model.

The multiple fitting models may be models negotiated in advance by the first device and the second device, and the first device may use multiple data as training data to train the multiple fitting models until the training end condition is satisfied. The training end condition may be that the number of parameter iterations reaches the maximum number, or the loss value determined based on the loss function is less than a preset value. After finishing the training, the first device determines the target fitting model according to the loss value of each fitting model.

In specific implementation, the first device can use the fitting model with the smallest loss value as the target fitting model, or the fitting model with the loss value less than the preset value as the target fitting model, or the loss value tends to converge The fitting model of is used as the target fitting model.

In some possible implementations, the multiple fitting models may specifically be at least two of a linear model, a polynomial model, and a neural network model. Considering the applicable scope of the model, multiple fitting models may include linear models, polynomial models, and neural network models. In this way, by fitting more models, the fitting accuracy can be improved and the accuracy of data transmission can be improved.

In some possible implementation manners, when the first device sends the target fitting model to the second device, it may send the identification of the fitting model selected by the first device and the corresponding model parameters to the second device. Specifically, when the first device and the second device negotiate the model, the model ID can be agreed in advance. For example, the model ID of a linear model can be 1, the model ID of a polynomial model can be 2, and the model ID of a neural network model can be 3. Models can generally be characterized by functions, and model parameters are the parameters of function expressions.

Taking the linear model as an example, the function expression is y=ax+b. Among them, x is the independent variable, y is the function value, and a and b are the parameters of the linear model. When the target fitting model is a linear model, the first device may send the identification “1” of the fitting model and the model parameters “a” and “b” to the second device.

Sending model identifiers and parameters can further reduce the amount of transmitted data and the amount of stored data, reduce transmission resource overhead and storage resource overhead, and save costs.

In some possible implementations, when multiple pieces of data acquired by the first device are associated with other data, the first device may also determine the association relationship between the multiple pieces of data and other data, and the association relationship may be characterized by a relationship function. Based on this, the first device can send the relationship function associated with the target fitting model to the second device. In this way, the relationship function can be used to determine another target fitting model based on the target fitting model. At least one of the other data associated with multiple data can be restored.

By sending the above relational function instead of directly sending other data associated with multiple data, the amount of transmitted data and the amount of stored data is further reduced, the compression performance is improved, and the transmission pressure and storage pressure are reduced.

In some possible implementation manners, the first device may also send the difference value to the second device. The difference is a difference between the plurality of data acquired by the first device and the plurality of data restored by the first device based on the target fitting model. The difference may be used in combination with the target fitting model to restore at least one data of the plurality of data. Among them, the above difference is superimposed on the recovered data of the target fitting model to achieve lossless data recovery and improve the accuracy of data transmission.

In some possible implementation manners, the foregoing multiple data may be multiple time series data within a time window. Among them, the time window can be set according to actual needs, for example, it can be set to one minute. Time series data refers to time series data, that is, a series of data recorded in chronological order for the same indicator. In an example, if the time window is one minute and the collection period is one second, then 60 data can be collected in one time window.

In the second aspect, this application provides a data processing method. Specifically, the second device as the data receiver receives the target fitting model from the first device, the target fitting model is obtained by the first device based on the multiple data acquired, and the second device recovers according to the target fitting model At least one of the multiple data.

Compared with directly receiving multiple data, receiving the target fitting model and storing the target fitting model greatly reduce the pressure of data transmission and data storage. Moreover, the second device can choose to restore at least one of the multiple data, so that the individual needs of different users can be met.

In some possible implementation manners, the second device may also receive a relationship function associated with the target fitting model, and determine another target fitting model based on the relationship function and the target fitting model. In this way, the second device may determine another target fitting model based on the relationship function and the target fitting model. A target fitting model restores at least one data among other data associated with multiple data.

Compared with directly receiving multiple data of related indicators, receiving the relationship function associated with the target fitting model can further reduce the amount of transmitted data and the amount of stored data, thereby reducing the pressure of data transmission and data storage.

In some possible implementation manners, the second device may choose to perform lossless recovery or lossy recovery for data according to requirements. Among them, for data with higher data accuracy requirements, lossless recovery can be selected, and for data with relatively low data accuracy requirements, lossy recovery can be selected to save costs.

In specific implementation, the second device directly restores the data according to the target fitting model, which is a lossy restoration. When the second device also receives the difference value from the first device, the difference value is the difference value between the multiple data acquired by the first device and the multiple data recovered by the first device based on the target fitting model, The second device may restore at least one of the multiple data according to the target fitting model and the difference. Specifically, the second device may first perform data recovery according to the target fitting model, and then superimpose the above-mentioned difference on this basis to obtain the original data, thus realizing data lossless recovery.

In a third aspect, this application provides a data processing device, which includes:

Communication module, used to obtain multiple data;

A fitting module, configured to obtain a target fitting model based on the plurality of data;

The communication module is further configured to send the target fitting model to a second device, and the target fitting model is used to restore at least one data of the plurality of data.

In some possible implementation manners, the fitting module is specifically used for:

Model fitting is performed on multiple fitting models according to the multiple data, so as to select one fitting model from the multiple fitting models as the target fitting model.

In some possible implementations, the multiple fitting models include at least two of a linear model, a polynomial model, and a neural network model.

In some possible implementation manners, the communication module is specifically used for:

Send the identification of the fitting model selected by the fitting module and the corresponding model parameters to the second device.

In some possible implementation manners, the communication module is also used to:

The relationship function associated with the target fitting model is sent to the second device, where the relationship function is used to determine another target fitting model based on the target fitting model, and the other target fitting model is used for At least one of the other data associated with the plurality of data is restored.

Send a difference value to the second device, where the difference value is used in combination with the target fitting model to recover at least one of the multiple data, and the difference value is the multiple data obtained by the first device. The difference between the data and the multiple data recovered by the first device based on the target fitting model.

In some possible implementation manners, the plurality of data includes a plurality of time series data within a time window.

In a fourth aspect, the present application provides a data processing device, which includes:

A communication module, configured to receive a target fitting model from a first device, where the target fitting model is obtained by the first device based on a plurality of acquired data;

The restoration module is configured to restore at least one of the multiple data according to the target fitting model.

Receiving a relationship function associated with the target fitting model;

The device also includes:

A determining module, configured to determine another target fitting model according to the relationship function and the target fitting model;

The recovery module is also used for:

At least one of the other data associated with the plurality of data is restored according to the another target fitting model.

A communication module, configured to receive a difference value from a first device, where the difference value is the difference between the multiple data acquired by the first device and the multiple data recovered by the first device based on the target fitting model Difference

The recovery module is specifically used for:

At least one data of the plurality of data is restored according to the target fitting model and the difference value.

In a fifth aspect, the present application provides a device including a processor and a memory;

The processor is configured to execute instructions stored in the memory, so that the processor executes the data processing method according to the first aspect or the second aspect.

In a sixth aspect, the present application provides a computer-readable storage medium, including instructions, which when run on a device, cause the device to execute the data processing method as described in the first or second aspect.

In a seventh aspect, this application provides a computer program product containing instructions, which when run on a device, causes the device to execute the data processing method described in the first or second aspect.

On the basis of the implementation manners provided by the above aspects, this application can be further combined to provide more implementation manners.

Description of the drawings

In order to more clearly illustrate the technical methods of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments.

FIG. 1 is a schematic diagram of a system architecture of a data processing method provided by an embodiment of this application;

FIG. 2 is an interaction flowchart of a data processing method provided by an embodiment of this application;

FIG. 3 is a schematic diagram of a data processing method provided by an embodiment of the application;

FIG. 4 is a schematic diagram of a data processing method provided by an embodiment of the application;

FIG. 5 is a schematic diagram of a data processing method provided by an embodiment of this application;

FIG. 6 is a schematic structural diagram of a data processing device provided by an embodiment of this application;

FIG. 7 is a schematic structural diagram of a data processing device provided by an embodiment of the application;

FIG. 8 is a schematic structural diagram of a device provided by an embodiment of this application;

FIG. 9 is a schematic structural diagram of a device provided by an embodiment of the application.

Detailed ways

The embodiments of the present application will be described below in conjunction with the drawings.

The terms "first", "second", etc. in the specification and claims of the application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is merely a way of distinguishing objects with the same attributes in the description of the embodiments of the present application.

Figure 1 shows a possible application scenario of an embodiment of the present application. In this application scenario, the device 102 and the device 104 are network node devices. The device 102 may be a network node device that sends data, and the device 104 is a network node device that receives the data. The data sent by the device 102 may be data generated by the device 102 itself, or may be data obtained by the device 102 from other devices. In some implementations, the device 102 may also be used as a device for receiving data, and the device 104 may also be used as a device for sending data. This application only uses the device 102 to send data to the device 104 as an example for description, and does not constitute a limitation to the technical solution of the application.

As shown in Figure 1, the device 102 can be a mobile terminal device such as a smart phone, a smart bracelet, a tablet computer, etc. The device can monitor any one or more indicators in the position and heartbeat through a position sensor, a heartbeat sensor, etc. Monitoring data. In some embodiments, the device 102 may also be a device deployed with an application on the cloud, and the device may monitor any one or more indicators such as throughput and delay of the application to generate monitoring data. Of course, the device 102 may also be an end-side device of the Internet of things (IoT), such as a smart refrigerator, a smart light bulb, etc., and the device may monitor any one or more indicators such as temperature and brightness to generate monitoring data.

Correspondingly, the device 104 may be a data transfer device. For example, in a cloud service scenario, the device 104 may be an edge device of the cloud, such as a gateway device (gateway, GW), a switch device, and so on. In some embodiments, the device 104 may also be a data consuming device, such as a server or the like.

If the device 102 directly sends the collected data to the device 104, when the data is large, on the one hand, a larger bandwidth resource is required, and on the other hand, the device 104 is required to have sufficient storage space. Therefore, higher requirements are proposed for network transmission and storage. Require. Based on this, the present application provides a data processing method that supports the device 102 as a data sender to obtain multiple data first, and then obtain a target fitting model based on the multiple data, and send the target fitting model instead of directly sending the multiple data. Individual data, in this way, can reduce the amount of data sent, reduce the requirements for network transmission, and reduce transmission pressure. And the receiver can store the target fitting model, and when the data needs to be used, restore at least one of the multiple data based on the target fitting model, which also reduces the requirements for data storage and reduces the storage pressure. Improve storage performance.

In order to make the technical solution of the present application clearer and easier to understand, the data processing method provided by the embodiment of the present application will be introduced in detail from the perspective of interaction between the device 102 and the device 104.

Referring to the interaction flowchart of the data processing method shown in FIG. 2, the method includes:

S202: The device 102 obtains multiple pieces of data.

In a specific implementation, the device 102 may generate multiple pieces of data or obtain multiple pieces of data from other devices. Among them, multiple data acquired by the device 102 (for example, data generated by the device 102 itself, data acquired by the device 102 from other devices) may be time series data, that is, data of an indicator within a time period. Of course, the multiple pieces of data acquired by the device 102 may also be non-sequential data, such as location-related data.

In order to facilitate understanding, the following description is combined with specific examples.

In an example, the multiple data acquired by the device 102 may be that the device monitors one or more of the virtual machine's central processing unit (CPU) usage rate, memory (memory) usage rate and other indicators to obtain The CPU usage of the virtual machine in a period of time and/or the memory usage of the virtual machine in a period of time. Of course, the multiple pieces of data acquired by the device 102 may also be the device's monitoring of one or more of the application's throughput, delay and other indicators, and the acquired throughput and/or delay of the application in a period of time.

It should be noted that the CPU usage rate, memory usage rate, throughput, and time delay in a time period described in the embodiments of the present application specifically refer to the CPU usage rate, memory usage rate, throughput, and time delay corresponding to different time points in a time period. Time delay. Based on this, the device 102 and the device 104 that perform data interaction can negotiate data collection rules in advance. For example, when the data is time series data, the device 102 and the device 104 may pre-negotiate the data collection period, that is, the collection time interval. Of course, the device 102 may also send collection rules to the device 104 when sending data, such as the collection period of the sent data.

In another example, the multiple data acquired by the device 102 may also be data generated by monitoring indicators such as rainfall at different locations. This data is non-time series data, which can change with different geographic locations. Similar to time series data, the device 102 and the device 104 may negotiate the data collection location interval in advance. For example, the collection location interval can be measured by latitude and longitude, such as one longitude or one latitude. In some possible implementations, the collection location interval can also be measured by length and height, such as 5 kilometers (kilometer, km), and so on.

When the device 102 acquires multiple pieces of data, it often needs to transmit metadata describing the data when transmitting the data. Metadata is specifically data describing data, which can be at least one attribute of data. For example, for data such as CPU usage, its metadata may include indicator names, project identifiers of the projects to which they belong, service names, and so on. The metadata of multiple data corresponding to the same indicator is often the same. Therefore, compared with directly transmitting data and its metadata, obtaining multiple data can realize the aggregation of metadata, and only need to transmit one piece of metadata. . In this way, data compression can be achieved.

After acquiring multiple data, the device 102 can further process the multiple data, such as performing compression processing, which can reduce transmission overhead and storage overhead. In a specific implementation, the device 102 can acquire multiple data according to the pre-configured acquisition quantity N, that is, the device 102 can acquire N data at a time. When the data acquired by the device 102 is time series data, the device 102 may acquire multiple time series data within the time window Δt. Wherein, the acquisition number N or the time window Δt can be set according to an empirical value, which is not limited in the embodiment of the application.

The data in the embodiments of this application may be numerical values. In some cases, the data may also be characters, such as characters A, B, C, D, etc. that characterize the level.

S204: The device 102 obtains a target fitting model based on the multiple data.

In specific implementation, the device 102 may perform model fitting on multiple data, and determine a fitting model that meets a preset condition as a target fitting model. In some implementations, in order to reduce the amount of calculation for model fitting and improve the efficiency of model fitting, the device 102 and the device 104 may negotiate multiple fitting models in advance. In this way, the device 102 may perform the multiple fitting models based on multiple data. The model performs model fitting to select one fitting model from multiple fitting models as the target fitting model.

Among them, the multiple fitting models negotiated by the device 102 and the device 104 may be models that are easy to fit. In some implementations, the multiple fitting models may be at least two of a linear model, a polynomial model, and a neural network model. Among them, the neural network model may be a three-layer neural network model. In this way, the complexity of the model can be reduced and the model is easier to fit.

When the device 102 simulates and fits multiple fitting models based on multiple data, it can calculate the loss value of each fitting model based on the multiple data and the data obtained from the model fitting, and then calculate the loss value of each fitting model according to the loss value from the multiple fitting models Choose a fitting model as the target fitting model.

Specifically, the device 102 may mark multiple pieces of data as training data, and then train a fitting model based on the training data in combination with a loss function, and optimize the fitting model parameters until the training end condition is satisfied. The training end condition may be that the number of parameter iterations reaches the maximum number, or the loss value determined based on the loss function is less than a preset value. After finishing the training, the device 102 determines the target fitting model according to the loss value of each fitting model.

In some possible implementations, the device 102 may use the fitting model with the smallest loss value as the target fitting model, or the fitting model with the loss value less than a preset value as the target fitting model, or the loss value tends to be The fitting model that converges is used as the target fitting model.

S206: The device 102 determines the difference between the multiple pieces of data it acquires and the multiple pieces of data recovered based on the target fitting model.

Specifically, the multiple data acquired by the device 102 have a one-to-one correspondence with the multiple data recovered based on the target fitting model. For example, if the device 102 acquires N pieces of data, it can restore the N pieces of data based on the target fitting model. The device 102 can make a difference between the data recovered by the corresponding target fitting model for each of the multiple acquired data, thereby obtaining multiple differences. The differences are used to achieve data lossless recovery, that is, according to the target model. The combined model and the difference realize the accurate recovery of multiple data acquired by the first device.

It should be understood that, when executing the data processing method provided in the embodiment of the present application, the device 102 may not execute the foregoing S206. For example, in a scenario where data lossy compression can meet the demand, the device 102 may not perform the above S206.

S208: The device 102 sends the target fitting model to the device 104.

In a specific implementation, the device 102 and the device 104 may agree on the model identifiers of the multiple fitting models negotiated. In this way, when the device 102 selects a fitting model from a plurality of fitting models as the target fitting model, it can send the identification of the fitted model selected and the corresponding model parameters to the device 104, so as to realize the sending of the target to the device 104 Fit the model. Based on this, the transmission overhead can be further reduced.

For ease of understanding, this application also provides a specific example for description.

In this example, the identification model_ID of the linear model may be 1, the identification model_ID of the polynomial model may be 2, and the identification model_ID of the neural network model may be 3. Among them, the linear model can be expressed as:

y=ax+b (1)

Among them, a and b are the parameters of the linear model, which identify the slope and intercept respectively. x can be time or latitude and longitude (representing a region), and y can be data restored by the device 104.

When the device 102 selects a linear model as the target fitting model, the device 102 may send a sequence or array including the following data, such as {1, a, b} or [1, a, b], to send the target to the device 104 Fit the model.

Assuming that the time window includes 60 pieces of data, compared with sending 60 pieces of data, in this embodiment of the present application, the device 102 only needs to send a very short sequence or array. For example, for a linear model, only the identification and model of the model need to be sent. There are three parameters in total, which is equivalent to 95% of the data compressed, which greatly reduces the amount of transmitted data and the amount of stored data.

It should be noted that the execution order of S206 and S208 may be arbitrary, for example, they may be executed in parallel, or may be executed sequentially in a set order.

S210: The device 102 sends to the device 104 the difference between the multiple pieces of data acquired by the device 102 and the multiple pieces of data restored based on the target fitting model.

When the device 102 sends the difference and the target fitting model, one message may be used to send, or multiple messages may be used to send separately.

The device 102 may execute the above S208 and S210 in parallel, or may execute successively in a set sequence.

It should be understood that, similar to S206, when executing the data processing method provided in the embodiment of the present application, the device 102 may not execute the foregoing S210. For example, in a scenario where data lossy compression can meet the demand, the device 102 may not perform the foregoing S210.

S212: The device 104 restores multiple data according to the target fitting model and the difference.

For ease of understanding, the data acquired by the device 102 can be denoted as y ₀ , the data recovered by the device 102 based on the target fitting model is denoted as y ₁ , the data acquired by the device 102 and the data recovered by the device 102 based on the target fitting model y ₁ The difference is recorded as Δy.

When the device 104 receives the target fitting model and the difference Δy, it may first restore y ₁ according to the target fitting model. For example, for time series data, the device 104 may substitute time into the expression of the target fitting model to restore y ₁ . It should be noted that for ordered time series data, such as periodic time series data, the device 104 may substitute the corresponding time into the expression of the target fitting model according to the agreed collection period directly to restore y ₁ . For non-ordered time series data, the device 102 can also send the time to the device 104, such as "1, 2, 4, 5". In this way, the device 104 can substitute the time sent by the device 102 into the expression recovery of the target fitting model. ₁ .

Further, after y _{1 is} restored, the device 104 may superimpose y 1 with the corresponding difference Δy, thereby restoring y ₀ , achieving lossless data recovery.

It should be understood that, similar to S206 and 210, when the data processing method provided in the embodiment of the present application is executed, the device 104 may not perform superimposing y1 with the corresponding difference Δy in a scenario where data lossy compression can meet the demand. Steps to restore y _0. That is, the device 104 can recover multiple data according to the target fitting model.

Based on the above description, the embodiment of the present application provides a data processing method that supports the device 102 as the data sender to obtain multiple data first, and then obtain a target fitting model based on the multiple data, and send the target fitting model Instead of sending multiple data directly, in this way, the amount of data sent can be reduced, the requirements for network transmission, and the transmission pressure can be reduced. And the device 104 as the receiver can store the target fitting model, and when data is needed, restore multiple data based on the target fitting model, which also reduces the requirements for data storage, reduces storage pressure, and improves storage performance .

Further, the data processing method provided in this application also supports two methods of lossy and lossless compression for users to choose. For the lossy compression mode, the device 102 sends the target fitting model, and the device 104 restores the data acquired by the device 102 according to the target fitting model. For the lossless compression mode, the device 102 also sends the difference between the data acquired by the device 102 and the data recovered based on the target fitting model, and the device 104 accurately restores the data acquired by the device 102 according to the target fitting model and the aforementioned difference. In addition, since the difference value sent by the device 102 is generally a small value, the number of bits occupied by the encoding will be less than that of a large value. Therefore, the compression effect can also be obtained and the transmission and storage pressure can be reduced.

To facilitate understanding, the embodiment of the present application also provides a schematic diagram of a data processing method. As shown in FIG. 3, the device 102 can acquire multiple data in a time window, and then perform model fitting according to the multiple acquired data to obtain a target fitting model. It should be noted that FIG. 3 uses the data as the time series data and the target fitting model as the sine model for illustration. In other possible implementations, it may also be other types of data and other types of models, which is not limited in this embodiment.

Then, the device 102 may send the target fitting model and the difference between the acquired data and the target fitting model recovery data to the device 104. In this example, the difference is specifically [0.01, 0.12, 0.03,..., 0.06]. The device 104 can perform data loss recovery according to the target fitting model. When the data accuracy is relatively high, the device 104 may also superimpose the aforementioned difference on the result of the lossy recovery of the data, so as to realize the lossless recovery of the data.

Figure 3 is an example of collecting one data at each time, that is, each time stamp corresponds to a value. However, in many scenarios, the values of multiple indicators can be collected at a time. For example, the device 102 collects environmental information once every minute, and the environmental information may include two indicators of temperature and humidity.

The multiple indicators collected by the device 102 may be independent of each other, or may have an associated relationship. For indicators that have an association relationship, the data can be further compressed through association to further reduce transmission overhead and storage overhead.

Specifically, for multiple indicators having an association relationship, the device 102 may perform model fitting only on multiple data corresponding to one indicator. For the convenience of description, the index for model fitting can be called the basic index. The device 102 may determine the association relationship between other indicators among the multiple indicators and the foregoing basic indicators according to the data of the multiple indicators. Among them, the association can be characterized by the relation function. For example, when the relationship is linear, it can be characterized by a linear function.

Based on this, other indicators of the device 102 send the relationship function associated with the target fitting model to the device 104, and the device 104 may determine another target fitting model according to the relationship function and the target fitting model. Specifically, the device 104 substitutes the expression of the target fitting model into the above-mentioned relation function, thereby obtaining the expression of another target fitting model. Then, the device 104 may restore other multiple data associated with the multiple data according to another target fitting model, that is, multiple data corresponding to other indicators associated with the basic indicator. Wherein, the specific implementation of the device 104 restoring the associated multiple data according to another target fitting model is similar to that of restoring multiple data according to one target model. For details, please refer to the description of related content above, which will not be repeated here.

For ease of understanding, this application also provides a specific example to illustrate the process of restoring multiple pieces of associated data.

As shown in FIG. 4, for data including multiple indicators, the device 102 may determine the association relationship between the indicators according to multiple data corresponding to each indicator. In this example, the multiple data of the 8 indicators in the time window can be divided into two groups, the indicators within the group have an association relationship, and the indicators between the groups do not have an association relationship.

For indicators that do not have an association relationship, they can be compressed according to the data processing method provided in the embodiment shown in FIG. 2. In this example, the device 102 may perform model fitting for multiple data corresponding to one indicator in each group of indicators to obtain a corresponding target fitting model. Then the device 102 sends the target fitting model and the relationship function representing the association relationship to the device 104. As shown in FIG. 4, the device 102 sends two target fitting models and three relationship functions associated with each target fitting model. The device 104 can restore multiple data corresponding to the index according to the target fitting model. In addition, the device 104 may determine the target fitting model corresponding to the index having the correlation relationship according to the target fitting model and the relationship function, and restore multiple pieces of related data based on the target fitting model corresponding to the index having the correlation relationship.

In the following, the data processing method of the present application will be introduced by taking the data processing of the CPU usage rate and the memory usage rate as an example.

As shown in FIG. 5, the device 102 collects the CPU usage rate and the memory usage rate according to the collection period of 1 second. In this example, the device 102 needs to collect data within 10 minutes and send the data within the 10 minutes to the device 104. In the traditional method, each time the device 102 collects a piece of data, its data and corresponding metadata are sent to the device 104.

Referring to FIG. 5, the metadata sent by the device 102 includes the project identifier project_id of the project to which the CPU belongs, the cluster identifier cluster_id, the service name service_name, the service instance name service_instance_name, and the metric name metric_name. Since the data sent by the device 102 is time series data, it is necessary to send the time stamp timestamp and the data value value when sending data. Based on this, the data of an indicator at a point in time may be a data block composed of the above metadata and data. The device 102 sends 1 index of data within 10 minutes, that is, 600 data blocks are sent.

Among them, the metadata corresponding to an indicator at different times is often the same. By obtaining multiple data (such as caching multiple data), the metadata of multiple data can be aggregated. Therefore, the device 102 only needs to send one data block. The data part in the data block may include a start timestamp timestamp_from and an end timestamp timestamp_to. In addition, the data value is also changed from a single value to an array of values corresponding to multiple moments. Since there is no need to repeatedly send repeated metadata, the device 104 does not need to store the repeated metadata, thereby reducing transmission pressure and storage pressure.

Further, the device 102 can perform model fitting based on multiple acquired data, specifically based on the aggregated array as training data, and perform the respective pre-set fitting models, such as linear model, polynomial model, and neural network model. Perform fitting, calculate the loss value of each fitting model, and determine the target fitting model. The device 102 only needs to record the model identification and model parameters in the value field. In this way, the value field can be reduced from an array including 60 data to an array including 3 data, which further reduces the amount of data and reduces transmission pressure and storage pressure. The device 104 receives the above-mentioned data and stores it. Correspondingly, when the device 104 has a demand for some data of CPU usage, the device 104 can read the model identification and model parameters of the value field, restore the target fitting model, and then restore multiple CPU usage data based on the target fitting model At least one data in.

When there is an association between memory usage and CPU usage, the data to be sent can be further compressed. Specifically, for the memory usage rate of the associated index, the device 102 only needs to send a data block composed of simplified metadata and simplified data. As shown in Figure 5, the simplified metadata includes the indicator name metric_name, which is memory in this example, and other metadata of the data can be referred to the associated indicators, which are also called reference indicators. The simplified data includes a reference index (reference, ref) field and a value field. The field value of the reference index field is CPU, and the value field is used to identify the relationship function between the index and the reference index. The value field includes function identification and function It consists of several numerical values including parameters. In this way, the amount of data that needs to be transmitted for the associated indicators is greatly reduced, and the transmission pressure and storage pressure are reduced. Correspondingly, when the device 104 has requirements for some data of the memory usage rate, the device 104 can read the function identifier and function parameters of the value field, thereby restoring the relationship function between the memory usage rate and the CPU usage rate, based on the relationship function and The target fitting model of the CPU usage rate restores the target fitting model of the memory usage rate, and then restores at least one of the multiple memory usage rate data based on the target fitting model.

The data processing method provided by the embodiment of the present application is described above with reference to FIGS. 1 to 5, and the data processing apparatus and equipment provided by the embodiment of the present application are described below with reference to the accompanying drawings.

Referring to the schematic structural diagram of the data processing device shown in FIG. 6, the data processing device 300 includes:

The communication module 302 is used to obtain multiple data;

The fitting module 304 is configured to obtain a target fitting model based on the multiple data;

The communication module 302 is further configured to send the target fitting model to the second device 102, where the target fitting model is used to restore at least one data of the plurality of data.

In some possible implementation manners, the fitting module 304 is specifically configured to:

In some possible implementation manners, the communication module 302 is specifically configured to:

The identification of the fitting model selected by the fitting module 304 and the corresponding model parameters are sent to the second device.

In some possible implementation manners, the communication module 302 is further configured to:

The data processing device 300 according to the embodiment of the present application may correspond to the implementation of the method described in the embodiment of the present application, and the above and other operations and/or functions of each module in the data processing device 300 are to implement each method in FIG. 2 respectively. For the sake of brevity, the corresponding process will not be repeated here.

Next, referring to the schematic structural diagram of the data processing device shown in FIG. 7, the data processing device 400 includes:

The communication module 402 is configured to receive a target fitting model from a first device, where the target fitting model is obtained by the first device based on a plurality of acquired data;

The restoration module 404 is configured to restore at least one of the multiple data according to the target fitting model.

In some possible implementation manners, the communication module 402 is further configured to:

Receiving a relationship function associated with the target fitting model;

The device 400 further includes:

The recovery module 404 is also used for:

The communication module 402 is configured to receive a difference value from a first device, where the difference value is the plurality of data acquired by the first device and the plurality of data recovered by the first device based on the target fitting model The difference;

The recovery module 404 is specifically used for:

The data processing apparatus 400 according to the embodiment of the present application may correspond to the method described in the embodiment of the present application, and the above-mentioned and other operations and/or functions of each module in the data processing apparatus 400 are to implement each method in FIG. 2 respectively. For the sake of brevity, the corresponding process will not be repeated here.

Figures 8-9 also provide a device. The device 500 shown in FIG. 8 may be specifically used to implement the functions of the data processing apparatus 300 in the embodiment shown in FIG. 6, and the device 600 shown in FIG. 9 may be specifically used to implement the data processing apparatus in the embodiment shown in FIG. 400 features.

The device 500 includes a bus 501, a processor 502, a communication interface 503, and a memory 504. The processor 502, the memory 504, and the communication interface 503 communicate through a bus 501. The bus 501 may be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in FIG. 8, but it does not mean that there is only one bus or one type of bus. The communication interface 503 is used to communicate with the outside, such as acquiring multiple data, sending a target fitting model to the second device, and so on.

The processor 502 may be a central processing unit (CPU). The memory 504 may include a volatile memory (volatile memory), such as a random access memory (random access memory, RAM). The memory 504 may also include a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), flash memory, HDD or SSD.

The memory 504 stores executable code, and the processor 502 executes the executable code to execute the aforementioned data processing method.

Specifically, in the case of implementing the embodiment shown in FIG. 6 and each module described in the embodiment of FIG. 6 is realized by software, the software or program required to execute the function of the fitting module 304 in FIG. 6 The code is stored in the memory 504. The function of the communication module 302 is implemented through the communication interface 503. The processor 502 is configured to execute instructions in the memory 504 and execute a data processing method applied to the data processing device 300.

The device 600 includes a bus 601, a processor 602, a communication interface 603, and a memory 604. The processor 602, the memory 604, and the communication interface 603 communicate through a bus 601. When the device 600 implements the embodiment shown in FIG. 7 and each module described in the embodiment of FIG. 7 is implemented by software, the software or program code required to execute the function of the recovery module 404 in 7 is stored in In the memory 604. The function of the communication module 402 is implemented through the communication interface 603. The processor 602 is configured to execute instructions in the memory 604, and execute a data processing method applied to the data processing device 400.

The embodiment of the present application also provides a computer-readable storage medium, including instructions, which when run on a device, cause the device to execute the above-mentioned data processing method applied to the data processing apparatus 300 or the data processing apparatus 400.

The embodiments of the present application also provide a computer program product. When the computer program product is executed by a computer, the computer executes any one of the aforementioned data processing methods. The computer program product may be a software installation package. In the case where any of the aforementioned data processing methods needs to be used, the computer program product may be downloaded and executed on the computer.

Claims

A data processing method, characterized in that the method includes:

The first device acquires multiple data;

The first device obtains a target fitting model based on the plurality of data;

The first device sends the target fitting model to the second device, and the target fitting model is used to restore at least one data of the plurality of data.
The method according to claim 1, wherein the first device obtaining a target fitting model based on the plurality of data comprises:

Model fitting is performed on multiple fitting models according to the multiple data, so as to select one fitting model from the multiple fitting models as the target fitting model.
The method according to claim 2, wherein the multiple fitting models include at least two of a linear model, a polynomial model, and a neural network model.
The method according to claim 2, wherein the sending of the target fitting model by the first device to the second device comprises:

The first device sends the identification of the fitting model selected by the first device and the corresponding model parameters to the second device.
The method according to any one of claims 1 to 4, wherein the method further comprises:

The first device sends a relationship function associated with the target fitting model to the second device, where the relationship function is used to determine another target fitting model based on the target fitting model, and the other target The fitting model is used to restore at least one data among other data associated with the plurality of data.
The method according to any one of claims 1 to 4, wherein the method further comprises:

The first device sends a difference value to the second device, the difference value is used to restore at least one of the multiple data in combination with the target fitting model, and the difference value is the first Differences between the plurality of data acquired by the device and the plurality of data restored by the first device based on the target fitting model.
The method according to any one of claims 1 to 4, wherein the plurality of data includes a plurality of time series data within a time window.
A data processing method, characterized in that the method includes:

The second device receives a target fitting model from the first device, where the target fitting model is obtained by the first device based on a plurality of acquired data;

The second device restores at least one data of the plurality of data according to the target fitting model.
The method according to claim 8, wherein the method further comprises:

Receiving, by the second device, a relationship function associated with the target fitting model;

The second device determines another target fitting model according to the relationship function and the target fitting model;

The second device restores at least one of the other data associated with the plurality of data according to the another target fitting model.
The method according to claim 8 or 9, wherein the method further comprises:

The second device receives a difference value from the first device, where the difference value is the difference between the plurality of data acquired by the first device and the plurality of data recovered by the first device based on the target fitting model Difference

The second device restoring at least one of the multiple data according to the target fitting model includes:

The second device restores at least one data of the plurality of data according to the target fitting model and the difference value.
A data processing device, characterized in that the device comprises:

Communication module, used to obtain multiple data;

A fitting module, configured to obtain a target fitting model based on the plurality of data;

The communication module is further configured to send the target fitting model to a second device, and the target fitting model is used to restore at least one data of the plurality of data.
The device according to claim 11, wherein the fitting module is specifically configured to:

Model fitting is performed on multiple fitting models according to the multiple data, so as to select one fitting model from the multiple fitting models as the target fitting model.
The device according to claim 12, wherein the multiple fitting models include at least two of a linear model, a polynomial model, and a neural network model.
The device according to claim 12, wherein the communication module is specifically configured to:

Send the identification of the fitting model selected by the fitting module and the corresponding model parameters to the second device.
The device according to any one of claims 11 to 14, wherein the communication module is further configured to:

The relationship function associated with the target fitting model is sent to the second device, where the relationship function is used to determine another target fitting model based on the target fitting model, and the other target fitting model is used for At least one of the other data associated with the plurality of data is restored.
The device according to any one of claims 11 to 14, wherein the communication module is further configured to:

Send a difference value to the second device, where the difference value is used in combination with the target fitting model to recover at least one of the multiple data, and the difference value is the multiple data obtained by the first device. The difference between the data and the multiple data recovered by the first device based on the target fitting model.
The device according to any one of claims 11 to 14, wherein the multiple data includes multiple time series data within a time window.
A data processing device, characterized in that the device comprises:

A communication module, configured to receive a target fitting model from a first device, where the target fitting model is obtained by the first device based on a plurality of acquired data;

The restoration module is configured to restore at least one of the multiple data according to the target fitting model.
The device according to claim 18, wherein the communication module is further configured to:

Receiving a relationship function associated with the target fitting model;

The device also includes:

A determining module, configured to determine another target fitting model according to the relationship function and the target fitting model;

The recovery module is also used for:

At least one of the other data associated with the plurality of data is restored according to the another target fitting model.
The device according to claim 18 or 19, wherein the communication module is further configured to:

A communication module, configured to receive a difference value from a first device, where the difference value is the difference between the multiple data acquired by the first device and the multiple data recovered by the first device based on the target fitting model Difference

The recovery module is specifically used for:

At least one data of the plurality of data is restored according to the target fitting model and the difference value.
A device, characterized in that the device includes a processor and a memory;

The processor is configured to execute instructions stored in the memory, so that the processor executes the method according to any one of claims 1 to 10.
A computer-readable storage medium, comprising instructions, which when run on a device, cause the device to execute the method according to any one of claims 1 to 10.